[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:05:21 dist-test-slave-xv68 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:05:21 dist-test-slave-xv68 krb5kdc[395](info): set up 2 sockets
Jun 20 03:05:21 dist-test-slave-xv68 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 03:05:22.718829 388 mini_kdc.cc:121] Time spent starting KDC: real 2.261s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 03:05:23.013959 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.295s user 0.000s sys 0.003s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 03:05:23.273108 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.259s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 03:05:23.441298 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.168s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/test-user.keytab.
W20260620 03:05:23.655167 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.214s user 0.000s sys 0.004s
Jun 20 03:05:23 dist-test-slave-xv68 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:05:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:05:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:05:23.726856 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46519
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:24.043526 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:24.043805 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:24.043891 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:24.051107 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:05:24.051200 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:24.051245 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:24.051316 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:05:24.051415 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:05:24.060784 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:24.062038 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:24.063336 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:24.074913 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:24.074965 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:24.075002 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:24.075716 408 server_base.cc:1034] running on GCE node
I20260620 03:05:24.076447 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:24.077768 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:24.078984 408 hybrid_clock.cc:648] HybridClock initialized: now 1781924724078921 us; error 79 us; skew 500 ppm
I20260620 03:05:24.079370 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:24 dist-test-slave-xv68 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:24.082362 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:05:24.084208 408 webserver.cc:460] Webserver started at http://127.0.97.62:45107/ using document root <none> and password file <none>
I20260620 03:05:24.084842 408 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:24.084932 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:24.085176 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:24.088076 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "27ab9b4ae56c4442b3a7ee185667a202"
format_stamp: "Formatted at 2026-06-20 03:05:24 on dist-test-slave-xv68"
server_key: "70309da91cc38d8a3e59d6d21029f4cc"
server_key_iv: "5cb54eaedd6e0c248d101b8006744b0b"
server_key_version: "encryptionkey@0"
I20260620 03:05:24.088752 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "27ab9b4ae56c4442b3a7ee185667a202"
format_stamp: "Formatted at 2026-06-20 03:05:24 on dist-test-slave-xv68"
server_key: "70309da91cc38d8a3e59d6d21029f4cc"
server_key_iv: "5cb54eaedd6e0c248d101b8006744b0b"
server_key_version: "encryptionkey@0"
I20260620 03:05:24.093641 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:05:24.096781 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:24.098003 408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 03:05:24.098140 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "27ab9b4ae56c4442b3a7ee185667a202"
format_stamp: "Formatted at 2026-06-20 03:05:24 on dist-test-slave-xv68"
server_key: "70309da91cc38d8a3e59d6d21029f4cc"
server_key_iv: "5cb54eaedd6e0c248d101b8006744b0b"
server_key_version: "encryptionkey@0"
I20260620 03:05:24.098287 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:24.129832 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:24.134002 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:24.134232 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:24.153225 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46519
I20260620 03:05:24.153252 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46519 every 8 connection(s)
I20260620 03:05:24.154484 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:05:24.158033 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:24.161609 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 408
I20260620 03:05:24.161916 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:05:24.164314 388 external_mini_cluster.cc:1428] Setting key 5a1ab78336e9a7a01473fcf83a03dee6
I20260620 03:05:24.170197 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202: Bootstrap starting.
I20260620 03:05:24.173089 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:24.174078 475 log.cc:824] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:24.176764 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202: No bootstrap required, opened a new log
I20260620 03:05:24.186380 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ab9b4ae56c4442b3a7ee185667a202" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46519 } }
I20260620 03:05:24.186839 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:24.186952 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ab9b4ae56c4442b3a7ee185667a202, State: Initialized, Role: FOLLOWER
I20260620 03:05:24.187659 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ab9b4ae56c4442b3a7ee185667a202" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46519 } }
I20260620 03:05:24.187933 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:05:24.188069 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:05:24.188263 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:24.191339 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ab9b4ae56c4442b3a7ee185667a202" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46519 } }
I20260620 03:05:24.191941 475 leader_election.cc:302] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27ab9b4ae56c4442b3a7ee185667a202; no voters:
I20260620 03:05:24.192533 475 leader_election.cc:288] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:05:24.192724 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:05:24.193958 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202 [term 1 LEADER]: Becoming Leader. State: Replica: 27ab9b4ae56c4442b3a7ee185667a202, State: Running, Role: LEADER
I20260620 03:05:24.194473 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ab9b4ae56c4442b3a7ee185667a202" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46519 } }
I20260620 03:05:24.194931 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 03:05:24 dist-test-slave-xv68 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:24.197156 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27ab9b4ae56c4442b3a7ee185667a202. Latest consensus state: current_term: 1 leader_uuid: "27ab9b4ae56c4442b3a7ee185667a202" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ab9b4ae56c4442b3a7ee185667a202" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46519 } } }
I20260620 03:05:24.197506 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202 [sys.catalog]: This master's current role is: LEADER
I20260620 03:05:24.197962 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27ab9b4ae56c4442b3a7ee185667a202" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ab9b4ae56c4442b3a7ee185667a202" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46519 } } }
I20260620 03:05:24.198210 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202 [sys.catalog]: This master's current role is: LEADER
I20260620 03:05:24.201313 486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:05:24.207660 486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:05:24.217078 486 catalog_manager.cc:1347] Generated new cluster ID: b88e092941a643c1bcaa15e73d7fa0ae
I20260620 03:05:24.217181 486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:05:24.218119 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:24.168709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.0.97.62:46519)
0620 03:05:24.169540 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:24.169555 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:24.171283 (+ 1728us) server_negotiation.cc:407] Connection header received
0620 03:05:24.181982 (+ 10699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:24.182021 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:24.182610 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:24.183135 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:24.186059 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:24.188061 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:24.190041 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:24.191033 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:24.191216 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:24.199117 (+ 7901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:24.199131 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:24.199156 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:24.199268 (+ 112us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:24.207610 (+ 8342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:24.208933 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:24.208949 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:24.208961 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:24.209241 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:24.209932 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:24.209940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:24.209946 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:24.210819 (+ 873us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:24.215045 (+ 4226us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:24.215396 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 03:05:24.215962 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":370,"threads_started":1}
I20260620 03:05:24.238749 486 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:05:24.240389 486 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:05:24.250484 486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27ab9b4ae56c4442b3a7ee185667a202: Generated new TSK 0
I20260620 03:05:24.251468 486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:05:24.303882 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46519
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:24.633803 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:24.634102 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:24.634195 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:24.641691 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:24.641789 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:24.641898 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:05:24.651010 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46519
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:24.652231 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:24.653452 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:24.666877 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:24.666910 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:24.666966 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:24.667708 501 server_base.cc:1034] running on GCE node
I20260620 03:05:24.668305 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:24.669270 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:24.670513 501 hybrid_clock.cc:648] HybridClock initialized: now 1781924724670454 us; error 54 us; skew 500 ppm
I20260620 03:05:24.670871 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:24 dist-test-slave-xv68 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:24.674011 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:05:24.675464 501 webserver.cc:460] Webserver started at http://127.0.97.1:34165/ using document root <none> and password file <none>
I20260620 03:05:24.676154 501 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:24.676280 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:24.676597 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:24.679793 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "542da7403d814efba3089afce2562d31"
format_stamp: "Formatted at 2026-06-20 03:05:24 on dist-test-slave-xv68"
server_key: "32e1fc86807bc13eb2dac67e9ce680d7"
server_key_iv: "b36eb91927648148cad82848dece2033"
server_key_version: "encryptionkey@0"
I20260620 03:05:24.680591 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "542da7403d814efba3089afce2562d31"
format_stamp: "Formatted at 2026-06-20 03:05:24 on dist-test-slave-xv68"
server_key: "32e1fc86807bc13eb2dac67e9ce680d7"
server_key_iv: "b36eb91927648148cad82848dece2033"
server_key_version: "encryptionkey@0"
I20260620 03:05:24.685740 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:05:24.689054 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:24.690361 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 03:05:24.690527 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "542da7403d814efba3089afce2562d31"
format_stamp: "Formatted at 2026-06-20 03:05:24 on dist-test-slave-xv68"
server_key: "32e1fc86807bc13eb2dac67e9ce680d7"
server_key_iv: "b36eb91927648148cad82848dece2033"
server_key_version: "encryptionkey@0"
I20260620 03:05:24.690692 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:24.702553 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:24.707284 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:24.707587 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:24.708801 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:05:24.710014 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:24.710083 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:24.710158 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:24.710199 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:24.742098 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35019
I20260620 03:05:24.742132 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35019 every 8 connection(s)
I20260620 03:05:24.743343 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:05:24.752034 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 501
I20260620 03:05:24.752249 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:05:24.752980 388 external_mini_cluster.cc:1428] Setting key 18cbd6acaa51eb1498f0ec54b6ccaafd
Jun 20 03:05:24 dist-test-slave-xv68 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:24.764074 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:24.745828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53859 (local address 127.0.97.62:46519)
0620 03:05:24.746255 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:24.746260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:24.747168 (+ 908us) server_negotiation.cc:407] Connection header received
0620 03:05:24.748986 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:24.748991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:24.749079 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:24.749196 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:24.750618 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:24.751395 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:24.753014 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:24.753458 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:24.753584 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:24.758065 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:24.758074 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:24.758081 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:24.758126 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:24.760901 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:24.761811 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:24.761815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:24.761817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:24.761944 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:24.762430 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:24.762433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:24.762435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:24.762670 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:24.762773 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:24.763676 (+ 903us) server_negotiation.cc:299] Negotiation successful
0620 03:05:24.763831 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":202,"threads_started":1}
I20260620 03:05:24.765610 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:24.746359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46519 (local address 127.0.97.1:53859)
0620 03:05:24.747020 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:24.747060 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:24.748732 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:24.749427 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:24.749436 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:24.749907 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:24.750442 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:24.750459 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:24.751548 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:24.751560 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:24.752873 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:24.752886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:24.753675 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:24.753682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:24.753933 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:24.754610 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:24.754630 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:24.757836 (+ 3206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:24.761027 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:24.761039 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:24.761058 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:24.761677 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:24.762067 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:24.762074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:24.762081 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:24.762325 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:24.762786 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:24.762796 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:24.763250 (+ 454us) client_negotiation.cc:769] Sending connection context
0620 03:05:24.763589 (+ 339us) client_negotiation.cc:240] Negotiation successful
0620 03:05:24.764150 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":261,"threads_started":1}
I20260620 03:05:24.767227 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46519
I20260620 03:05:24.767547 629 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:24.768225 629 heartbeater.cc:511] Master 127.0.97.62:46519 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 03:05:24.770320 440 ts_manager.cc:194] Registered new tserver with Master: 542da7403d814efba3089afce2562d31 (127.0.97.1:35019)
I20260620 03:05:24.771876 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:05:24.794397 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46519
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:25.143916 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:25.144224 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:25.144414 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:25.151827 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:25.151968 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:25.152156 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:05:25.161711 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46519
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:25.163057 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:25.164680 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:25.182324 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:25.182399 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:25.182570 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:25.183115 635 server_base.cc:1034] running on GCE node
I20260620 03:05:25.183804 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:25.184830 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:25.186079 635 hybrid_clock.cc:648] HybridClock initialized: now 1781924725186011 us; error 63 us; skew 500 ppm
I20260620 03:05:25.186448 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:25 dist-test-slave-xv68 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:25.189659 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:05:25.191078 635 webserver.cc:460] Webserver started at http://127.0.97.2:42611/ using document root <none> and password file <none>
I20260620 03:05:25.191826 635 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:25.191924 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:25.192183 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:25.195259 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "710bca46c8694cb6878c951693f81cf9"
format_stamp: "Formatted at 2026-06-20 03:05:25 on dist-test-slave-xv68"
server_key: "665f43e0def8083dd5e5e525b6dcd236"
server_key_iv: "e6fdc22d62f8e34a3bb039b760a24a2f"
server_key_version: "encryptionkey@0"
I20260620 03:05:25.196174 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "710bca46c8694cb6878c951693f81cf9"
format_stamp: "Formatted at 2026-06-20 03:05:25 on dist-test-slave-xv68"
server_key: "665f43e0def8083dd5e5e525b6dcd236"
server_key_iv: "e6fdc22d62f8e34a3bb039b760a24a2f"
server_key_version: "encryptionkey@0"
I20260620 03:05:25.202067 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:05:25.205811 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:25.207188 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 03:05:25.207444 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "710bca46c8694cb6878c951693f81cf9"
format_stamp: "Formatted at 2026-06-20 03:05:25 on dist-test-slave-xv68"
server_key: "665f43e0def8083dd5e5e525b6dcd236"
server_key_iv: "e6fdc22d62f8e34a3bb039b760a24a2f"
server_key_version: "encryptionkey@0"
I20260620 03:05:25.207675 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:25.223572 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:25.227512 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:25.227797 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:25.228742 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:05:25.230023 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:25.230125 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:25.230266 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:25.230335 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:25.264113 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44873
I20260620 03:05:25.264194 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44873 every 8 connection(s)
I20260620 03:05:25.265578 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:05:25.270900 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 635
I20260620 03:05:25.271099 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:05:25.271581 388 external_mini_cluster.cc:1428] Setting key 4c7569caf4d22217ffcfcf0f9cf6f81c
Jun 20 03:05:25 dist-test-slave-xv68 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:25.286543 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:25.268268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51097 (local address 127.0.97.62:46519)
0620 03:05:25.268720 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:25.268724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:25.269553 (+ 829us) server_negotiation.cc:407] Connection header received
0620 03:05:25.271581 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:25.271587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:25.271675 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:25.271823 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:25.273231 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:25.273930 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:25.275633 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:25.276072 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:25.276227 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:25.280541 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:25.280546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:25.280550 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:25.280579 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:25.283161 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:25.284163 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:25.284167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:25.284169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:25.284296 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:25.284813 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:25.284817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:25.284819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:25.285053 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:25.285166 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:25.286109 (+ 943us) server_negotiation.cc:299] Negotiation successful
0620 03:05:25.286269 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":262,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 03:05:25.288060 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:25.268740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46519 (local address 127.0.97.2:51097)
0620 03:05:25.269394 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:25.269437 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:25.271273 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:25.272062 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:25.272070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:25.272547 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:25.273073 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:25.273088 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:25.274079 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:25.274084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:25.275276 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:25.275342 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:25.276325 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:25.276333 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:25.276588 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:25.277283 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:25.277320 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:25.280300 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:25.283330 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:25.283343 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:25.283365 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:25.284019 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:25.284416 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:25.284423 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:25.284429 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:25.284697 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:25.285185 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:25.285193 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:25.285654 (+ 461us) client_negotiation.cc:769] Sending connection context
0620 03:05:25.286073 (+ 419us) client_negotiation.cc:240] Negotiation successful
0620 03:05:25.286657 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":257,"threads_started":1}
I20260620 03:05:25.289453 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46519
I20260620 03:05:25.289762 763 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 03:05:25.290603 763 heartbeater.cc:511] Master 127.0.97.62:46519 requested a full tablet report, sending...
I20260620 03:05:25.292085 440 ts_manager.cc:194] Registered new tserver with Master: 710bca46c8694cb6878c951693f81cf9 (127.0.97.2:44873)
I20260620 03:05:25.292852 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:05:25.310585 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46519
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:25.630642 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:25.630944 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:25.631119 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:25.638764 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:25.638901 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:25.639087 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:05:25.648715 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46519
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:25.649938 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:25.651243 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:25.664393 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:25.664559 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:25.664430 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:25.665205 769 server_base.cc:1034] running on GCE node
I20260620 03:05:25.665838 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:25.666730 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:25.667982 769 hybrid_clock.cc:648] HybridClock initialized: now 1781924725667940 us; error 49 us; skew 500 ppm
I20260620 03:05:25.668318 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:25 dist-test-slave-xv68 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:25.671268 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:05:25.673210 769 webserver.cc:460] Webserver started at http://127.0.97.3:33167/ using document root <none> and password file <none>
I20260620 03:05:25.673878 769 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:25.674019 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:25.674316 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:25.677165 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "cffc8fbae9484f3b8d52ec2eef822e96"
format_stamp: "Formatted at 2026-06-20 03:05:25 on dist-test-slave-xv68"
server_key: "d0a0365f814678cf1d0574a6566df471"
server_key_iv: "a1c7c912c099b290c487d5482a399c2b"
server_key_version: "encryptionkey@0"
I20260620 03:05:25.677945 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cffc8fbae9484f3b8d52ec2eef822e96"
format_stamp: "Formatted at 2026-06-20 03:05:25 on dist-test-slave-xv68"
server_key: "d0a0365f814678cf1d0574a6566df471"
server_key_iv: "a1c7c912c099b290c487d5482a399c2b"
server_key_version: "encryptionkey@0"
I20260620 03:05:25.682910 769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 03:05:25.686669 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:25.687987 769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 03:05:25.688162 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "cffc8fbae9484f3b8d52ec2eef822e96"
format_stamp: "Formatted at 2026-06-20 03:05:25 on dist-test-slave-xv68"
server_key: "d0a0365f814678cf1d0574a6566df471"
server_key_iv: "a1c7c912c099b290c487d5482a399c2b"
server_key_version: "encryptionkey@0"
I20260620 03:05:25.688366 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:25.721983 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:25.726450 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:25.726725 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:25.727752 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:05:25.729023 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:25.729120 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:25.729272 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:25.729344 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:25.765302 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40459
I20260620 03:05:25.765457 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40459 every 8 connection(s)
I20260620 03:05:25.766551 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:05:25.774843 629 heartbeater.cc:503] Master 127.0.97.62:46519 was elected leader, sending a full tablet report...
I20260620 03:05:25.775686 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 769
I20260620 03:05:25.775866 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:05:25.776369 388 external_mini_cluster.cc:1428] Setting key fa8a1c75ab6c52e5372f5e8c7c47de5b
Jun 20 03:05:25 dist-test-slave-xv68 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:25.786693 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:25.769035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35393 (local address 127.0.97.62:46519)
0620 03:05:25.769211 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:25.769216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:25.770777 (+ 1561us) server_negotiation.cc:407] Connection header received
0620 03:05:25.772599 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:25.772604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:25.772706 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:25.772817 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:25.774206 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:25.775146 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:25.777385 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:25.777789 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:25.777937 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:25.781605 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:25.781610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:25.781613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:25.781643 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:25.784172 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:25.784804 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:25.784808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:25.784810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:25.784917 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:25.785327 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:25.785331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:25.785333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:25.785593 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:25.785694 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:25.786295 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 03:05:25.786453 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:05:25.787688 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:25.769521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46519 (local address 127.0.97.3:35393)
0620 03:05:25.770615 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:25.770658 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:25.772329 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:25.773042 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:25.773050 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:25.773525 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:25.774041 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:25.774058 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:25.775327 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:25.775333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:25.777247 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:25.777258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:25.778250 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:25.778261 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:25.778503 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:25.779151 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:25.779171 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:25.781405 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:25.784286 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:25.784293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:25.784305 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:25.784697 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:25.785032 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:25.785036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:25.785039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:25.785231 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:25.785691 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:25.785696 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:25.785988 (+ 292us) client_negotiation.cc:769] Sending connection context
0620 03:05:25.786212 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 03:05:25.786570 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":668,"threads_started":1}
I20260620 03:05:25.788982 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46519
I20260620 03:05:25.789274 897 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:25.789937 897 heartbeater.cc:511] Master 127.0.97.62:46519 requested a full tablet report, sending...
I20260620 03:05:25.791648 439 ts_manager.cc:194] Registered new tserver with Master: cffc8fbae9484f3b8d52ec2eef822e96 (127.0.97.3:40459)
I20260620 03:05:25.792488 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:05:25.792749 439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35393
Jun 20 03:05:25 dist-test-slave-xv68 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924725, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 03:05:25 dist-test-slave-xv68 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924725, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:05:25.835870 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:25.821242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.0.97.1:35019)
0620 03:05:25.821863 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:25.821870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:25.821897 (+ 27us) server_negotiation.cc:407] Connection header received
0620 03:05:25.821956 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:25.821962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:25.822167 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:25.822290 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:25.823717 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:25.824794 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:25.826012 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:25.826571 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:25.826666 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:25.830702 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:25.830713 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:25.830727 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:25.830761 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:25.833499 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:25.834070 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:25.834077 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:25.834079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:25.834189 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:25.834509 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:25.834511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:25.834513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:25.834820 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:25.835018 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:25.835241 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 03:05:25.835479 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":335,"threads_started":1}
W20260620 03:05:25.837085 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48654
Jun 20 03:05:25 dist-test-slave-xv68 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 03:05:25 dist-test-slave-xv68 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:05:25.880738 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:25.867341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48666 (local address 127.0.97.1:35019)
0620 03:05:25.867553 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:25.867559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:25.867784 (+ 225us) server_negotiation.cc:407] Connection header received
0620 03:05:25.867996 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:25.868000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:25.868068 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:25.868173 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:25.869442 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:25.870109 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:25.871328 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:25.871810 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:25.871944 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:25.875912 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:25.875916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:25.875920 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:25.875951 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:25.878231 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:25.878883 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:25.878893 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:25.878895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:25.879029 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:25.879401 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:25.879406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:25.879411 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:25.879644 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:25.879886 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:25.880190 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 03:05:25.880418 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 03:05:25.882180 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 501
I20260620 03:05:25.893652 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:25.883009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48670 (local address 127.0.97.1:35019)
0620 03:05:25.883152 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:25.883156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:25.883627 (+ 471us) server_negotiation.cc:407] Connection header received
0620 03:05:25.883728 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:25.883731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:25.883781 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:25.883857 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:25.884909 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:25.885568 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:25.886687 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:25.887095 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:25.887186 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:25.889215 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:25.889220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:25.889223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:25.889248 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:25.891548 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:25.892141 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:25.892163 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:25.892169 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:25.892304 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:25.892589 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:25.892591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:25.892593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:25.892803 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:25.893038 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:25.893204 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 03:05:25.893338 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 03:05:25.894292 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:05:25.976179 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 501
I20260620 03:05:25.992141 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 635
Jun 20 03:05:25 dist-test-slave-xv68 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:05:26.007946 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:25.993013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49268 (local address 127.0.97.2:44873)
0620 03:05:25.993583 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:25.993590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:25.993614 (+ 24us) server_negotiation.cc:407] Connection header received
0620 03:05:25.993675 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:25.993681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:25.993878 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:25.994058 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:25.995225 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:25.996349 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:25.997582 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:25.998106 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:25.998245 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:26.002521 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:26.002532 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:26.002547 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:26.002613 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:26.005494 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:26.006067 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:26.006074 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:26.006076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:26.006186 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:26.006538 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:26.006542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:26.006544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:26.006870 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:26.007099 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:26.007446 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 03:05:26.007663 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":302,"threads_started":1}
I20260620 03:05:26.009282 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:05:26.091506 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 635
I20260620 03:05:26.113953 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 769
Jun 20 03:05:26 dist-test-slave-xv68 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:05:26.129007 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:26.114921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49028 (local address 127.0.97.3:40459)
0620 03:05:26.115629 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:26.115636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:26.115664 (+ 28us) server_negotiation.cc:407] Connection header received
0620 03:05:26.115723 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:26.115729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:26.115926 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:26.116049 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:26.117197 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:26.118213 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:26.119514 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:26.120025 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:26.120128 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:26.124129 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:26.124138 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:26.124151 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:26.124180 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:26.126648 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:26.127240 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:26.127247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:26.127249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:26.127427 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:26.127736 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:26.127739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:26.127741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:26.128032 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:26.128259 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:26.128464 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 03:05:26.128635 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":396,"threads_started":1}
I20260620 03:05:26.130203 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:05:26.211275 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 769
I20260620 03:05:26.227065 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 408
I20260620 03:05:26.228298 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:05:26.328128 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 408
2026-06-20T03:05:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5922 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:05:26 dist-test-slave-xv68 krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:05:26 dist-test-slave-xv68 krb5kdc[918](info): set up 2 sockets
Jun 20 03:05:26 dist-test-slave-xv68 krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260620 03:05:27.427249 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:05:27 dist-test-slave-xv68 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:05:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:05:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:05:27.596844 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36927
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:27.906690 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:27.906953 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:27.907122 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:27.914120 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:05:27.914211 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:27.914302 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:27.914403 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:05:27.914465 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:05:27.923666 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:27.924930 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:27.926203 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:27.937419 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:27.937343 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:27.937393 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:27.938200 931 server_base.cc:1034] running on GCE node
I20260620 03:05:27.938921 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:27.940259 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:27.941452 931 hybrid_clock.cc:648] HybridClock initialized: now 1781924727941398 us; error 39 us; skew 500 ppm
I20260620 03:05:27.941738 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:27 dist-test-slave-xv68 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:27.944679 931 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:05:27.945993 931 webserver.cc:460] Webserver started at http://127.0.97.62:35341/ using document root <none> and password file <none>
I20260620 03:05:27.946599 931 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:27.946696 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:27.946935 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:27.949762 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "9a5dc852cee6417eb31c9b3e84cc1c2c"
format_stamp: "Formatted at 2026-06-20 03:05:27 on dist-test-slave-xv68"
server_key: "666ea66007b4cf7148c32f916f452c32"
server_key_iv: "aaa73680ef94ff6c820c63947ce11ec4"
server_key_version: "encryptionkey@0"
I20260620 03:05:27.950435 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "9a5dc852cee6417eb31c9b3e84cc1c2c"
format_stamp: "Formatted at 2026-06-20 03:05:27 on dist-test-slave-xv68"
server_key: "666ea66007b4cf7148c32f916f452c32"
server_key_iv: "aaa73680ef94ff6c820c63947ce11ec4"
server_key_version: "encryptionkey@0"
I20260620 03:05:27.955237 931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260620 03:05:27.958506 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:27.959831 931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:05:27.959965 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "9a5dc852cee6417eb31c9b3e84cc1c2c"
format_stamp: "Formatted at 2026-06-20 03:05:27 on dist-test-slave-xv68"
server_key: "666ea66007b4cf7148c32f916f452c32"
server_key_iv: "aaa73680ef94ff6c820c63947ce11ec4"
server_key_version: "encryptionkey@0"
I20260620 03:05:27.960135 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:27.974160 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:27.977955 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:27.978183 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:27.995931 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36927
I20260620 03:05:27.996024 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36927 every 8 connection(s)
I20260620 03:05:27.997259 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:05:28.000823 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:28.002260 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 931
I20260620 03:05:28.002487 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:05:28.003322 388 external_mini_cluster.cc:1428] Setting key 4c448c4a2d9ee55b62e905bb456f0618
I20260620 03:05:28.014891 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c: Bootstrap starting.
Jun 20 03:05:28 dist-test-slave-xv68 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:28.018353 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:28.019368 998 log.cc:824] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:28.021940 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c: No bootstrap required, opened a new log
I20260620 03:05:28.025782 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:28.005335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48102 (local address 127.0.97.62:36927)
0620 03:05:28.006127 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:28.006138 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:28.006184 (+ 46us) server_negotiation.cc:407] Connection header received
0620 03:05:28.007790 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:28.007810 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:28.008179 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:28.008479 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:28.010140 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:28.011476 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:28.013193 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:28.013806 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:28.013922 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:28.018300 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:28.018308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:28.018320 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:28.018358 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:28.021499 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:28.022154 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:28.022161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:28.022169 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:28.022371 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:28.022721 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:28.022726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:28.022729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:28.023399 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:28.023713 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:28.024050 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 03:05:28.024457 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":383,"threads_started":1}
I20260620 03:05:28.030417 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5dc852cee6417eb31c9b3e84cc1c2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36927 } }
I20260620 03:05:28.030711 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:28.030781 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a5dc852cee6417eb31c9b3e84cc1c2c, State: Initialized, Role: FOLLOWER
I20260620 03:05:28.031232 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5dc852cee6417eb31c9b3e84cc1c2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36927 } }
I20260620 03:05:28.031531 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:05:28.031675 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:05:28.031843 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:28.033843 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5dc852cee6417eb31c9b3e84cc1c2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36927 } }
I20260620 03:05:28.034304 998 leader_election.cc:302] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a5dc852cee6417eb31c9b3e84cc1c2c; no voters:
I20260620 03:05:28.034845 998 leader_election.cc:288] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:05:28.035010 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:05:28.035698 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c [term 1 LEADER]: Becoming Leader. State: Replica: 9a5dc852cee6417eb31c9b3e84cc1c2c, State: Running, Role: LEADER
I20260620 03:05:28.036181 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5dc852cee6417eb31c9b3e84cc1c2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36927 } }
I20260620 03:05:28.036970 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:05:28.038913 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a5dc852cee6417eb31c9b3e84cc1c2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5dc852cee6417eb31c9b3e84cc1c2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36927 } } }
I20260620 03:05:28.039379 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c [sys.catalog]: This master's current role is: LEADER
I20260620 03:05:28.039252 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a5dc852cee6417eb31c9b3e84cc1c2c. Latest consensus state: current_term: 1 leader_uuid: "9a5dc852cee6417eb31c9b3e84cc1c2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5dc852cee6417eb31c9b3e84cc1c2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36927 } } }
I20260620 03:05:28.039527 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c [sys.catalog]: This master's current role is: LEADER
I20260620 03:05:28.041529 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:05:28.045890 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:05:28.053766 1011 catalog_manager.cc:1347] Generated new cluster ID: 079cbbc610e04be9a44bd31f90e31221
I20260620 03:05:28.053867 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:05:28.063365 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:05:28.064656 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:05:28.070820 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a5dc852cee6417eb31c9b3e84cc1c2c: Generated new TSK 0
I20260620 03:05:28.071580 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:05:28.116171 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36927
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:28.432132 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:28.432451 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:28.432621 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:28.439937 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:28.440086 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:28.440256 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:05:28.450142 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36927
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:28.451546 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:28.452817 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:28.465546 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:28.465579 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:28.465497 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:28.466966 1024 server_base.cc:1034] running on GCE node
I20260620 03:05:28.467548 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:28.468439 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:28.469641 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781924728469577 us; error 53 us; skew 500 ppm
I20260620 03:05:28.469964 1024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:28 dist-test-slave-xv68 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:28.473222 1024 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:05:28.474592 1024 webserver.cc:460] Webserver started at http://127.0.97.1:46187/ using document root <none> and password file <none>
I20260620 03:05:28.475248 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:28.475409 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:28.475692 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:28.478461 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "b8270b8bf38e4de088fa03b3bf73496a"
format_stamp: "Formatted at 2026-06-20 03:05:28 on dist-test-slave-xv68"
server_key: "d0996a1cf6b5663f3c452270b955cdbd"
server_key_iv: "f1479b56a9b14614a0b0720a8d886d83"
server_key_version: "encryptionkey@0"
I20260620 03:05:28.479199 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b8270b8bf38e4de088fa03b3bf73496a"
format_stamp: "Formatted at 2026-06-20 03:05:28 on dist-test-slave-xv68"
server_key: "d0996a1cf6b5663f3c452270b955cdbd"
server_key_iv: "f1479b56a9b14614a0b0720a8d886d83"
server_key_version: "encryptionkey@0"
I20260620 03:05:28.484378 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 03:05:28.487763 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:28.489164 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:05:28.489351 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "b8270b8bf38e4de088fa03b3bf73496a"
format_stamp: "Formatted at 2026-06-20 03:05:28 on dist-test-slave-xv68"
server_key: "d0996a1cf6b5663f3c452270b955cdbd"
server_key_iv: "f1479b56a9b14614a0b0720a8d886d83"
server_key_version: "encryptionkey@0"
I20260620 03:05:28.489545 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:28.508435 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:28.512764 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:28.513044 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:28.514076 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:05:28.515373 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:28.515441 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:28.515612 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:28.515671 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:28.547708 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46141
I20260620 03:05:28.547746 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46141 every 8 connection(s)
I20260620 03:05:28.548952 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:05:28.552615 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 1024
I20260620 03:05:28.552838 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:05:28.553510 388 external_mini_cluster.cc:1428] Setting key fab34036dc9f4c15166f085a937fe797
Jun 20 03:05:28 dist-test-slave-xv68 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 03:05:28.570226 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:28.551534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57603 (local address 127.0.97.62:36927)
0620 03:05:28.551964 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:28.551969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:28.553106 (+ 1137us) server_negotiation.cc:407] Connection header received
0620 03:05:28.554883 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:28.554888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:28.554957 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:28.555083 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:28.556733 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:28.557892 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:28.559437 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:28.559812 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:28.559908 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:28.564160 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:28.564165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:28.564168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:28.564201 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:28.567049 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:28.567840 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:28.567848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:28.567856 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:28.567962 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:28.568391 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:28.568395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:28.568398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:28.568735 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:28.568883 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:28.569632 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 03:05:28.569840 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":236,"threads_started":1}
I20260620 03:05:28.571404 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:28.552128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36927 (local address 127.0.97.1:57603)
0620 03:05:28.552929 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:28.552977 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:28.554638 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:28.555382 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:28.555390 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:28.555999 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:28.556555 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:28.556572 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:28.558038 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:28.558042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:28.559229 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:28.559240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:28.560040 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:28.560048 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:28.560372 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:28.561088 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:28.561136 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:28.563915 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:28.567202 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:28.567211 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:28.567229 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:28.567713 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:28.568081 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:28.568086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:28.568089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:28.568273 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:28.568860 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:28.568869 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:28.569226 (+ 357us) client_negotiation.cc:769] Sending connection context
0620 03:05:28.569554 (+ 328us) client_negotiation.cc:240] Negotiation successful
0620 03:05:28.570080 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":325,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 03:05:28.573010 1152 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36927
I20260620 03:05:28.573346 1152 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:28.574241 1152 heartbeater.cc:511] Master 127.0.97.62:36927 requested a full tablet report, sending...
I20260620 03:05:28.576750 963 ts_manager.cc:194] Registered new tserver with Master: b8270b8bf38e4de088fa03b3bf73496a (127.0.97.1:46141)
I20260620 03:05:28.578159 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:05:28.594511 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36927
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:28.905812 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:28.906081 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:28.906250 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:28.913242 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:28.913337 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:28.913528 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:05:28.922987 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36927
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:28.924288 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:28.925632 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:28.938614 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:28.938643 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:28.938880 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:28.939498 1158 server_base.cc:1034] running on GCE node
I20260620 03:05:28.940058 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:28.940948 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:28.942179 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781924728942129 us; error 63 us; skew 500 ppm
I20260620 03:05:28.942519 1158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:28 dist-test-slave-xv68 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:28.945834 1158 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:05:28.947420 1158 webserver.cc:460] Webserver started at http://127.0.97.2:44881/ using document root <none> and password file <none>
I20260620 03:05:28.948091 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:28.948236 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:28.948551 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:28.951436 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "e67b6f0a74834f63adb5b1f9d2bf20d1"
format_stamp: "Formatted at 2026-06-20 03:05:28 on dist-test-slave-xv68"
server_key: "c42c94698a9138ad9d535b9082d35796"
server_key_iv: "ea9cef4d6fb320248008427105c85bfb"
server_key_version: "encryptionkey@0"
I20260620 03:05:28.952235 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e67b6f0a74834f63adb5b1f9d2bf20d1"
format_stamp: "Formatted at 2026-06-20 03:05:28 on dist-test-slave-xv68"
server_key: "c42c94698a9138ad9d535b9082d35796"
server_key_iv: "ea9cef4d6fb320248008427105c85bfb"
server_key_version: "encryptionkey@0"
I20260620 03:05:28.957147 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:05:28.960438 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:28.961730 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:05:28.961889 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "e67b6f0a74834f63adb5b1f9d2bf20d1"
format_stamp: "Formatted at 2026-06-20 03:05:28 on dist-test-slave-xv68"
server_key: "c42c94698a9138ad9d535b9082d35796"
server_key_iv: "ea9cef4d6fb320248008427105c85bfb"
server_key_version: "encryptionkey@0"
I20260620 03:05:28.962075 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:28.987275 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:28.991375 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:28.991698 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:28.992693 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:05:28.994071 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:28.994169 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:28.994314 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:28.994385 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:29.027606 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32881
I20260620 03:05:29.027693 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32881 every 8 connection(s)
I20260620 03:05:29.028868 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:05:29.030031 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 1158
I20260620 03:05:29.030225 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:05:29.030746 388 external_mini_cluster.cc:1428] Setting key ee06be43a0bb1287b77971baa8f97dbc
Jun 20 03:05:29 dist-test-slave-xv68 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 03:05:29.050413 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.031706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35643 (local address 127.0.97.62:36927)
0620 03:05:29.031938 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:29.031944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:29.033140 (+ 1196us) server_negotiation.cc:407] Connection header received
0620 03:05:29.034906 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:29.034910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:29.035002 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:29.035111 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:29.036752 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.037532 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.040088 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.040466 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.040556 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.044785 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:29.044790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:29.044793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:29.044822 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:29.047488 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:29.048173 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:29.048183 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:29.048185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:29.048301 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:29.048799 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:29.048803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:29.048807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:29.049084 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:29.049233 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:29.049873 (+ 640us) server_negotiation.cc:299] Negotiation successful
0620 03:05:29.050074 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 03:05:29.051527 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.032286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36927 (local address 127.0.97.2:35643)
0620 03:05:29.033004 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:29.033041 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:29.034660 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:29.035400 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:29.035408 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:29.035903 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:29.036569 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:29.036597 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.037674 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.037682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:29.039959 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:29.039976 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.040679 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.040688 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:29.040997 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.041674 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:29.041705 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:29.044480 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:29.047612 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:29.047621 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:29.047638 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:29.048031 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:29.048426 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:29.048430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:29.048434 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:29.048680 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:29.049214 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:29.049221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:29.049530 (+ 309us) client_negotiation.cc:769] Sending connection context
0620 03:05:29.049777 (+ 247us) client_negotiation.cc:240] Negotiation successful
0620 03:05:29.050275 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":292,"threads_started":1}
I20260620 03:05:29.053021 1286 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36927
I20260620 03:05:29.053352 1286 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:29.054172 1286 heartbeater.cc:511] Master 127.0.97.62:36927 requested a full tablet report, sending...
I20260620 03:05:29.055576 963 ts_manager.cc:194] Registered new tserver with Master: e67b6f0a74834f63adb5b1f9d2bf20d1 (127.0.97.2:32881)
I20260620 03:05:29.056339 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:05:29.070559 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36927
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:29.388892 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:29.389163 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:29.389339 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:29.397096 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:29.397229 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:29.397461 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:05:29.408684 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36927
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:29.409919 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:29.411171 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:29.424206 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:29.424245 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:29.424628 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:29.424901 1291 server_base.cc:1034] running on GCE node
I20260620 03:05:29.425487 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:29.426396 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:29.427625 1291 hybrid_clock.cc:648] HybridClock initialized: now 1781924729427583 us; error 36 us; skew 500 ppm
I20260620 03:05:29.427927 1291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:29 dist-test-slave-xv68 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:29.430889 1291 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:05:29.432328 1291 webserver.cc:460] Webserver started at http://127.0.97.3:40797/ using document root <none> and password file <none>
I20260620 03:05:29.432976 1291 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:29.433115 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:29.433411 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:29.436286 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "e57ff67e83174681a4f43643b46800b5"
format_stamp: "Formatted at 2026-06-20 03:05:29 on dist-test-slave-xv68"
server_key: "5f7e7590c580488d6fad00ded8456c15"
server_key_iv: "0a4ec2e532066a9cf3e5d5b8b80dd7e4"
server_key_version: "encryptionkey@0"
I20260620 03:05:29.437074 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e57ff67e83174681a4f43643b46800b5"
format_stamp: "Formatted at 2026-06-20 03:05:29 on dist-test-slave-xv68"
server_key: "5f7e7590c580488d6fad00ded8456c15"
server_key_iv: "0a4ec2e532066a9cf3e5d5b8b80dd7e4"
server_key_version: "encryptionkey@0"
I20260620 03:05:29.442004 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:05:29.445459 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:29.446774 1291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:05:29.446951 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "e57ff67e83174681a4f43643b46800b5"
format_stamp: "Formatted at 2026-06-20 03:05:29 on dist-test-slave-xv68"
server_key: "5f7e7590c580488d6fad00ded8456c15"
server_key_iv: "0a4ec2e532066a9cf3e5d5b8b80dd7e4"
server_key_version: "encryptionkey@0"
I20260620 03:05:29.447140 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:29.464900 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:29.469194 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:29.469470 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:29.470377 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:05:29.471715 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:29.471812 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:29.471946 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:29.472014 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:29.505654 1291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40269
I20260620 03:05:29.505741 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40269 every 8 connection(s)
I20260620 03:05:29.506971 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:05:29.516968 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 1291
I20260620 03:05:29.517164 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:05:29.517669 388 external_mini_cluster.cc:1428] Setting key 75545fbaefaa62a745872af4f26f463f
Jun 20 03:05:29 dist-test-slave-xv68 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:29.526486 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.509645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56489 (local address 127.0.97.62:36927)
0620 03:05:29.509847 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:29.509852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:29.510946 (+ 1094us) server_negotiation.cc:407] Connection header received
0620 03:05:29.512761 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:29.512765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:29.512826 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:29.512915 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:29.514317 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.515086 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.516944 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.517435 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.517530 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.521324 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:29.521328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:29.521331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:29.521359 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:29.523849 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:29.524515 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:29.524521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:29.524524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:29.524650 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:29.525071 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:29.525074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:29.525077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:29.525353 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:29.525461 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:29.526106 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 03:05:29.526255 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 03:05:29.527453 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.510080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36927 (local address 127.0.97.3:56489)
0620 03:05:29.510787 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:29.510827 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:29.512508 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:29.513117 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:29.513124 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:29.513623 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:29.514161 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:29.514177 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.515253 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.515257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:29.516782 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:29.516799 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.517639 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.517646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:29.517927 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.518664 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:29.518686 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:29.521146 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:29.523963 (+ 2817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:29.523969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:29.523980 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:29.524374 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:29.524757 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:29.524761 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:29.524763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:29.524930 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:29.525454 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:29.525459 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:29.525771 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 03:05:29.525981 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 03:05:29.526317 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":287,"threads_started":1}
I20260620 03:05:29.528787 1419 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36927
I20260620 03:05:29.529068 1419 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:29.529744 1419 heartbeater.cc:511] Master 127.0.97.62:36927 requested a full tablet report, sending...
I20260620 03:05:29.531244 963 ts_manager.cc:194] Registered new tserver with Master: e57ff67e83174681a4f43643b46800b5 (127.0.97.3:40269)
I20260620 03:05:29.532415 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56489
I20260620 03:05:29.533006 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:05:29.552090 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.539052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48108 (local address 127.0.97.62:36927)
0620 03:05:29.539254 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:29.539262 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:29.539720 (+ 458us) server_negotiation.cc:407] Connection header received
0620 03:05:29.539838 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:29.539842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:29.539895 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:29.539977 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:29.541761 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.542469 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.543842 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.544272 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.544373 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.546581 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:29.546585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:29.546589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:29.546615 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:29.549259 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:29.549969 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:29.549973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:29.549975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:29.550079 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:29.550535 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:29.550538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:29.550540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:29.550888 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:29.551161 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:29.551546 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 03:05:29.551772 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:05:29.558058 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:05:29.565160 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:05:29.581166 1152 heartbeater.cc:503] Master 127.0.97.62:36927 was elected leader, sending a full tablet report...
I20260620 03:05:29.589995 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.576822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32881 (local address 127.0.0.1:47856)
0620 03:05:29.577670 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:29.577719 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:29.577923 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:29.579309 (+ 1386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:29.579320 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:29.579361 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:29.579862 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:29.579878 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.582627 (+ 2749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.582636 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:29.585906 (+ 3270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:29.585928 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.588813 (+ 2885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.588824 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:29.589125 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.589207 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 03:05:29.589334 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 03:05:29.589547 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":369,"threads_started":1}
I20260620 03:05:29.591710 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.576977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.0.97.2:32881)
0620 03:05:29.578518 (+ 1541us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:29.578531 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:29.578581 (+ 50us) server_negotiation.cc:407] Connection header received
0620 03:05:29.578695 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:29.578706 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:29.579004 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:29.580101 (+ 1097us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:29.580402 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.582412 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.586186 (+ 3774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.588108 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.590677 (+ 2569us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.590970 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:29.591067 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 03:05:29.591327 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1370,"thread_start_us":1267,"threads_started":1}
I20260620 03:05:29.595755 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 39d08ee848d14ffb858d7269ba520062 (DEFAULT_TABLE table=test-table [id=d8942a3bfa114cd4aeaf24f09e1efcaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:29.597589 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d08ee848d14ffb858d7269ba520062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:29.598420 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.577343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40269 (local address 127.0.0.1:53804)
0620 03:05:29.590305 (+ 12962us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:29.590329 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:29.590462 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:29.592455 (+ 1993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:29.592462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:29.592493 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:29.592652 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:29.592663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.594711 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.594718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:29.596051 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:29.596062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.597659 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.597665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:29.597830 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.597861 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 03:05:29.597952 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 03:05:29.598092 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12837,"mutex_wait_us":671,"thread_start_us":8315,"threads_started":1}
I20260620 03:05:29.599112 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.577757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53804 (local address 127.0.97.3:40269)
0620 03:05:29.589677 (+ 11920us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:29.589686 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:29.591910 (+ 2224us) server_negotiation.cc:407] Connection header received
0620 03:05:29.592014 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:29.592022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:29.592260 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:29.592828 (+ 568us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:29.593096 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.594563 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.596194 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.597241 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.597367 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.597588 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:29.598631 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0620 03:05:29.598792 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11745,"thread_start_us":11619,"threads_started":1}
I20260620 03:05:29.600704 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.577966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46141 (local address 127.0.0.1:48262)
0620 03:05:29.586587 (+ 8621us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:29.586606 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:29.586727 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:29.592622 (+ 5895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:29.592627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:29.592647 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:29.593068 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:29.593081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.594911 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.594916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:29.596902 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:29.596916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.599969 (+ 3053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.599977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:29.600098 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.600120 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 03:05:29.600232 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 03:05:29.600353 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8481,"thread_start_us":13510,"threads_started":1}
I20260620 03:05:29.600751 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.578429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48262 (local address 127.0.97.1:46141)
0620 03:05:29.591781 (+ 13352us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:29.591790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:29.591824 (+ 34us) server_negotiation.cc:407] Connection header received
0620 03:05:29.591924 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:29.591932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:29.592188 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:29.592396 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:29.593264 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.594773 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.597074 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.598977 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.599119 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.599362 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:29.600279 (+ 917us) server_negotiation.cc:299] Negotiation successful
0620 03:05:29.600421 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13184,"thread_start_us":13181,"threads_started":1}
I20260620 03:05:29.602613 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 39d08ee848d14ffb858d7269ba520062 (DEFAULT_TABLE table=test-table [id=d8942a3bfa114cd4aeaf24f09e1efcaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:29.604070 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d08ee848d14ffb858d7269ba520062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:29.603958 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 39d08ee848d14ffb858d7269ba520062 (DEFAULT_TABLE table=test-table [id=d8942a3bfa114cd4aeaf24f09e1efcaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:29.605077 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d08ee848d14ffb858d7269ba520062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:29.612287 1437 tablet_bootstrap.cc:492] T 39d08ee848d14ffb858d7269ba520062 P e67b6f0a74834f63adb5b1f9d2bf20d1: Bootstrap starting.
I20260620 03:05:29.615965 1437 tablet_bootstrap.cc:654] T 39d08ee848d14ffb858d7269ba520062 P e67b6f0a74834f63adb5b1f9d2bf20d1: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:29.617445 1437 log.cc:824] T 39d08ee848d14ffb858d7269ba520062 P e67b6f0a74834f63adb5b1f9d2bf20d1: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:29.617777 1438 tablet_bootstrap.cc:492] T 39d08ee848d14ffb858d7269ba520062 P e57ff67e83174681a4f43643b46800b5: Bootstrap starting.
I20260620 03:05:29.620986 1438 tablet_bootstrap.cc:654] T 39d08ee848d14ffb858d7269ba520062 P e57ff67e83174681a4f43643b46800b5: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:29.621330 1437 tablet_bootstrap.cc:492] T 39d08ee848d14ffb858d7269ba520062 P e67b6f0a74834f63adb5b1f9d2bf20d1: No bootstrap required, opened a new log
I20260620 03:05:29.621577 1437 ts_tablet_manager.cc:1397] T 39d08ee848d14ffb858d7269ba520062 P e67b6f0a74834f63adb5b1f9d2bf20d1: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 03:05:29.621958 1438 log.cc:824] T 39d08ee848d14ffb858d7269ba520062 P e57ff67e83174681a4f43643b46800b5: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:29.622551 1439 tablet_bootstrap.cc:492] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a: Bootstrap starting.
I20260620 03:05:29.625097 1438 tablet_bootstrap.cc:492] T 39d08ee848d14ffb858d7269ba520062 P e57ff67e83174681a4f43643b46800b5: No bootstrap required, opened a new log
I20260620 03:05:29.625344 1438 ts_tablet_manager.cc:1397] T 39d08ee848d14ffb858d7269ba520062 P e57ff67e83174681a4f43643b46800b5: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 03:05:29.626345 1439 tablet_bootstrap.cc:654] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:29.627488 1439 log.cc:824] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:29.629628 1437 raft_consensus.cc:348] T 39d08ee848d14ffb858d7269ba520062 P e67b6f0a74834f63adb5b1f9d2bf20d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67b6f0a74834f63adb5b1f9d2bf20d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32881 } } peers { permanent_uuid: "b8270b8bf38e4de088fa03b3bf73496a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46141 } } peers { permanent_uuid: "e57ff67e83174681a4f43643b46800b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } }
I20260620 03:05:29.629977 1437 raft_consensus.cc:374] T 39d08ee848d14ffb858d7269ba520062 P e67b6f0a74834f63adb5b1f9d2bf20d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:29.630064 1437 raft_consensus.cc:729] T 39d08ee848d14ffb858d7269ba520062 P e67b6f0a74834f63adb5b1f9d2bf20d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e67b6f0a74834f63adb5b1f9d2bf20d1, State: Initialized, Role: FOLLOWER
I20260620 03:05:29.630668 1439 tablet_bootstrap.cc:492] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a: No bootstrap required, opened a new log
I20260620 03:05:29.630627 1437 consensus_queue.cc:260] T 39d08ee848d14ffb858d7269ba520062 P e67b6f0a74834f63adb5b1f9d2bf20d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67b6f0a74834f63adb5b1f9d2bf20d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32881 } } peers { permanent_uuid: "b8270b8bf38e4de088fa03b3bf73496a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46141 } } peers { permanent_uuid: "e57ff67e83174681a4f43643b46800b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } }
I20260620 03:05:29.630910 1439 ts_tablet_manager.cc:1397] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20260620 03:05:29.631742 1286 heartbeater.cc:503] Master 127.0.97.62:36927 was elected leader, sending a full tablet report...
I20260620 03:05:29.632287 1437 ts_tablet_manager.cc:1428] T 39d08ee848d14ffb858d7269ba520062 P e67b6f0a74834f63adb5b1f9d2bf20d1: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260620 03:05:29.635116 1438 raft_consensus.cc:348] T 39d08ee848d14ffb858d7269ba520062 P e57ff67e83174681a4f43643b46800b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67b6f0a74834f63adb5b1f9d2bf20d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32881 } } peers { permanent_uuid: "b8270b8bf38e4de088fa03b3bf73496a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46141 } } peers { permanent_uuid: "e57ff67e83174681a4f43643b46800b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } }
I20260620 03:05:29.635548 1438 raft_consensus.cc:374] T 39d08ee848d14ffb858d7269ba520062 P e57ff67e83174681a4f43643b46800b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:29.635659 1438 raft_consensus.cc:729] T 39d08ee848d14ffb858d7269ba520062 P e57ff67e83174681a4f43643b46800b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57ff67e83174681a4f43643b46800b5, State: Initialized, Role: FOLLOWER
I20260620 03:05:29.636273 1438 consensus_queue.cc:260] T 39d08ee848d14ffb858d7269ba520062 P e57ff67e83174681a4f43643b46800b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67b6f0a74834f63adb5b1f9d2bf20d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32881 } } peers { permanent_uuid: "b8270b8bf38e4de088fa03b3bf73496a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46141 } } peers { permanent_uuid: "e57ff67e83174681a4f43643b46800b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } }
I20260620 03:05:29.637466 1419 heartbeater.cc:503] Master 127.0.97.62:36927 was elected leader, sending a full tablet report...
I20260620 03:05:29.638113 1438 ts_tablet_manager.cc:1428] T 39d08ee848d14ffb858d7269ba520062 P e57ff67e83174681a4f43643b46800b5: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 03:05:29.641469 1439 raft_consensus.cc:348] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67b6f0a74834f63adb5b1f9d2bf20d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32881 } } peers { permanent_uuid: "b8270b8bf38e4de088fa03b3bf73496a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46141 } } peers { permanent_uuid: "e57ff67e83174681a4f43643b46800b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } }
I20260620 03:05:29.641921 1439 raft_consensus.cc:374] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:29.642047 1439 raft_consensus.cc:729] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8270b8bf38e4de088fa03b3bf73496a, State: Initialized, Role: FOLLOWER
I20260620 03:05:29.642590 1439 consensus_queue.cc:260] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67b6f0a74834f63adb5b1f9d2bf20d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32881 } } peers { permanent_uuid: "b8270b8bf38e4de088fa03b3bf73496a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46141 } } peers { permanent_uuid: "e57ff67e83174681a4f43643b46800b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } }
I20260620 03:05:29.644112 1439 ts_tablet_manager.cc:1428] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260620 03:05:29.757177 1445 raft_consensus.cc:482] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:05:29.757510 1445 raft_consensus.cc:504] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67b6f0a74834f63adb5b1f9d2bf20d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32881 } } peers { permanent_uuid: "b8270b8bf38e4de088fa03b3bf73496a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46141 } } peers { permanent_uuid: "e57ff67e83174681a4f43643b46800b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } }
W20260620 03:05:29.758594 1420 tablet.cc:2369] T 39d08ee848d14ffb858d7269ba520062 P e57ff67e83174681a4f43643b46800b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:05:29.759390 1445 leader_election.cc:288] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e67b6f0a74834f63adb5b1f9d2bf20d1 (127.0.97.2:32881), e57ff67e83174681a4f43643b46800b5 (127.0.97.3:40269)
I20260620 03:05:29.766520 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.759873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35391 (local address 127.0.97.3:40269)
0620 03:05:29.760045 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:29.760062 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:29.760633 (+ 571us) server_negotiation.cc:407] Connection header received
0620 03:05:29.760861 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:29.760865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:29.760917 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:29.761026 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:29.762067 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.762788 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.764738 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.765536 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.765625 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.765744 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:29.766122 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 03:05:29.766243 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 03:05:29.766527 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.759897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35755 (local address 127.0.97.2:32881)
0620 03:05:29.760073 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:29.760080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:29.760601 (+ 521us) server_negotiation.cc:407] Connection header received
0620 03:05:29.760809 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:29.760815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:29.760889 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:29.760953 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:29.762100 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.762770 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.764658 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.765570 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.765716 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.765801 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:29.766144 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 03:05:29.766265 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:05:29.766682 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.759586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32881 (local address 127.0.97.1:35755)
0620 03:05:29.760474 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:29.760503 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:29.760658 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:29.761007 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:29.761012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:29.761034 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:29.761929 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:29.761939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.762893 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.762898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:29.764514 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:29.764529 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.765806 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.765811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:29.765968 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.765995 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 03:05:29.766109 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 03:05:29.766298 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":101,"thread_start_us":513,"threads_started":1}
I20260620 03:05:29.767066 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.759772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40269 (local address 127.0.97.1:35391)
0620 03:05:29.760503 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:29.760518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:29.760711 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:29.761050 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:29.761055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:29.761078 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:29.761922 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:29.761932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.762943 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.762947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:29.764594 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:29.764610 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.765720 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.765726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:29.765898 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.765941 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 03:05:29.766066 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 03:05:29.766271 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":9,"spinlock_wait_cycles":17792,"thread_start_us":475,"threads_started":1}
I20260620 03:05:29.767680 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d08ee848d14ffb858d7269ba520062" candidate_uuid: "b8270b8bf38e4de088fa03b3bf73496a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e67b6f0a74834f63adb5b1f9d2bf20d1" is_pre_election: true
I20260620 03:05:29.768010 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d08ee848d14ffb858d7269ba520062" candidate_uuid: "b8270b8bf38e4de088fa03b3bf73496a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57ff67e83174681a4f43643b46800b5" is_pre_election: true
I20260620 03:05:29.768241 1241 raft_consensus.cc:2454] T 39d08ee848d14ffb858d7269ba520062 P e67b6f0a74834f63adb5b1f9d2bf20d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8270b8bf38e4de088fa03b3bf73496a in term 0.
I20260620 03:05:29.768458 1374 raft_consensus.cc:2454] T 39d08ee848d14ffb858d7269ba520062 P e57ff67e83174681a4f43643b46800b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8270b8bf38e4de088fa03b3bf73496a in term 0.
I20260620 03:05:29.768987 1042 leader_election.cc:302] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b8270b8bf38e4de088fa03b3bf73496a, e57ff67e83174681a4f43643b46800b5, e67b6f0a74834f63adb5b1f9d2bf20d1; no voters:
I20260620 03:05:29.769345 1445 raft_consensus.cc:2781] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:05:29.769483 1445 raft_consensus.cc:482] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:05:29.769601 1445 raft_consensus.cc:3037] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:29.772176 1445 raft_consensus.cc:504] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67b6f0a74834f63adb5b1f9d2bf20d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32881 } } peers { permanent_uuid: "b8270b8bf38e4de088fa03b3bf73496a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46141 } } peers { permanent_uuid: "e57ff67e83174681a4f43643b46800b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } }
I20260620 03:05:29.772754 1445 leader_election.cc:288] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a [CANDIDATE]: Term 1 election: Requested vote from peers e67b6f0a74834f63adb5b1f9d2bf20d1 (127.0.97.2:32881), e57ff67e83174681a4f43643b46800b5 (127.0.97.3:40269)
I20260620 03:05:29.773104 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d08ee848d14ffb858d7269ba520062" candidate_uuid: "b8270b8bf38e4de088fa03b3bf73496a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e67b6f0a74834f63adb5b1f9d2bf20d1"
I20260620 03:05:29.773239 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d08ee848d14ffb858d7269ba520062" candidate_uuid: "b8270b8bf38e4de088fa03b3bf73496a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57ff67e83174681a4f43643b46800b5"
I20260620 03:05:29.773306 1241 raft_consensus.cc:3037] T 39d08ee848d14ffb858d7269ba520062 P e67b6f0a74834f63adb5b1f9d2bf20d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:29.773424 1374 raft_consensus.cc:3037] T 39d08ee848d14ffb858d7269ba520062 P e57ff67e83174681a4f43643b46800b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:29.776453 1241 raft_consensus.cc:2454] T 39d08ee848d14ffb858d7269ba520062 P e67b6f0a74834f63adb5b1f9d2bf20d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8270b8bf38e4de088fa03b3bf73496a in term 1.
I20260620 03:05:29.776453 1374 raft_consensus.cc:2454] T 39d08ee848d14ffb858d7269ba520062 P e57ff67e83174681a4f43643b46800b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8270b8bf38e4de088fa03b3bf73496a in term 1.
I20260620 03:05:29.776868 1040 leader_election.cc:302] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8270b8bf38e4de088fa03b3bf73496a, e57ff67e83174681a4f43643b46800b5; no voters:
I20260620 03:05:29.777170 1445 raft_consensus.cc:2781] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:05:29.777601 1445 raft_consensus.cc:686] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a [term 1 LEADER]: Becoming Leader. State: Replica: b8270b8bf38e4de088fa03b3bf73496a, State: Running, Role: LEADER
I20260620 03:05:29.778041 1445 consensus_queue.cc:237] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67b6f0a74834f63adb5b1f9d2bf20d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32881 } } peers { permanent_uuid: "b8270b8bf38e4de088fa03b3bf73496a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46141 } } peers { permanent_uuid: "e57ff67e83174681a4f43643b46800b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } }
W20260620 03:05:29.780917 1287 tablet.cc:2369] T 39d08ee848d14ffb858d7269ba520062 P e67b6f0a74834f63adb5b1f9d2bf20d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:05:29.782223 963 catalog_manager.cc:5495] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a reported cstate change: term changed from 0 to 1, leader changed from <none> to b8270b8bf38e4de088fa03b3bf73496a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b8270b8bf38e4de088fa03b3bf73496a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67b6f0a74834f63adb5b1f9d2bf20d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8270b8bf38e4de088fa03b3bf73496a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e57ff67e83174681a4f43643b46800b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } health_report { overall_health: UNKNOWN } } }
W20260620 03:05:29.801285 1153 tablet.cc:2369] T 39d08ee848d14ffb858d7269ba520062 P b8270b8bf38e4de088fa03b3bf73496a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 03:05:29 dist-test-slave-xv68 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:05:29.838253 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.822873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48270 (local address 127.0.97.1:46141)
0620 03:05:29.823069 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:29.823076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:29.823599 (+ 523us) server_negotiation.cc:407] Connection header received
0620 03:05:29.823797 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:29.823809 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:29.823869 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:29.823974 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:29.825929 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.826673 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.828007 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.828400 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.828530 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.832518 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:29.832533 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:29.832549 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:29.832586 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:29.835504 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:29.836182 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:29.836190 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:29.836192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:29.836323 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:29.836716 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:29.836720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:29.836722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:29.837113 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:29.837397 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:29.837686 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 03:05:29.837911 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 03:05:29 dist-test-slave-xv68 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924729, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 03:05:29 dist-test-slave-xv68 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924729, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:05:29.883106 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.869251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.0.97.1:46141)
0620 03:05:29.869413 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:29.869418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:29.869837 (+ 419us) server_negotiation.cc:407] Connection header received
0620 03:05:29.869958 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:29.869963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:29.870031 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:29.870115 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:29.871404 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.872599 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.873823 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.874231 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.874331 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.878136 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:29.878141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:29.878144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:29.878178 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:29.880808 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:29.881384 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:29.881388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:29.881390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:29.881540 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:29.881907 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:29.881911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:29.881913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:29.882178 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:29.882375 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:29.882681 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 03:05:29.882852 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 03:05:29 dist-test-slave-xv68 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 03:05:29 dist-test-slave-xv68 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:05:29.926357 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.913029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48282 (local address 127.0.97.1:46141)
0620 03:05:29.913273 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:29.913279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:29.913532 (+ 253us) server_negotiation.cc:407] Connection header received
0620 03:05:29.913741 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:29.913745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:29.913829 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:29.913922 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:29.915230 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.916061 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.917410 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.917798 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.917893 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.921917 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:29.921921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:29.921925 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:29.921956 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:29.924340 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:29.924908 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:29.924911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:29.924913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:29.925020 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:29.925309 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:29.925312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:29.925314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:29.925542 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:29.925711 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:29.925942 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 03:05:29.926088 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 03:05:29.928031 1067 tablet_service.cc:2908] Rejecting scan request for tablet 39d08ee848d14ffb858d7269ba520062: Uninitialized: clean time has not yet been initialized
I20260620 03:05:29.930810 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 1024
I20260620 03:05:29.942730 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:29.931681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.0.97.1:46141)
0620 03:05:29.931858 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:29.931866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:29.932182 (+ 316us) server_negotiation.cc:407] Connection header received
0620 03:05:29.932294 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:29.932298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:29.932362 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:29.932438 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:29.933467 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.934146 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.935427 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:29.935938 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:29.936028 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:29.938145 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:29.938149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:29.938153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:29.938179 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:29.940587 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:29.941160 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:29.941164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:29.941166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:29.941279 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:29.941569 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:29.941572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:29.941574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:29.941792 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:29.941958 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:29.942200 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 03:05:29.942370 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 03:05:29.943392 1147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:05:30.032922 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 1024
I20260620 03:05:30.055668 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 1158
Jun 20 03:05:30 dist-test-slave-xv68 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:05:30.070704 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:30.056443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47870 (local address 127.0.97.2:32881)
0620 03:05:30.056628 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:30.056633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:30.056650 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:05:30.056795 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:30.056799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:30.056888 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:30.056967 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:30.058118 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:30.058898 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:30.060312 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:30.060710 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:30.060815 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:30.064920 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:30.064931 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:30.064947 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:30.064979 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:30.067843 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:30.068482 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:30.068490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:30.068492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:30.068682 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:30.069104 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:30.069109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:30.069111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:30.069466 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:30.069709 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:30.070181 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 03:05:30.070331 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 03:05:30.071403 1281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:05:30.159435 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 1158
I20260620 03:05:30.176286 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 1291
Jun 20 03:05:30 dist-test-slave-xv68 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:05:30.191045 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:30.177101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53820 (local address 127.0.97.3:40269)
0620 03:05:30.177304 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:30.177311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:30.177333 (+ 22us) server_negotiation.cc:407] Connection header received
0620 03:05:30.177518 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:30.177524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:30.177603 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:30.177716 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:30.178983 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:30.179797 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:30.181060 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:30.181452 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:30.181581 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:30.185787 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:30.185803 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:30.185820 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:30.185862 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:30.188419 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:30.189087 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:30.189095 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:30.189097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:30.189236 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:30.189595 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:30.189601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:30.189604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:30.189929 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:30.190212 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:30.190503 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 03:05:30.190712 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 03:05:30.191805 1414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:05:30.279054 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 1291
I20260620 03:05:30.294821 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 931
I20260620 03:05:30.295921 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:05:30.401495 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 931
2026-06-20T03:05:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4074 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:05:30 dist-test-slave-xv68 krb5kdc[1467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:05:30 dist-test-slave-xv68 krb5kdc[1467](info): set up 2 sockets
Jun 20 03:05:30 dist-test-slave-xv68 krb5kdc[1467](info): commencing operation
krb5kdc: starting...
W20260620 03:05:31.483479 388 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:05:31 dist-test-slave-xv68 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:05:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:05:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:05:31.657017 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40011
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:31.969264 1480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:31.969534 1480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:31.969707 1480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:31.976826 1480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:05:31.976933 1480 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 03:05:31.977017 1480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:31.977133 1480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:31.977221 1480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:05:31.977306 1480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:05:31.987021 1480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40011
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:31.988379 1480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:31.989751 1480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:32.001358 1485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:32.001381 1488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:32.001704 1486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:32.002221 1480 server_base.cc:1034] running on GCE node
I20260620 03:05:32.003039 1480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:32.004470 1480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:32.005717 1480 hybrid_clock.cc:648] HybridClock initialized: now 1781924732005646 us; error 51 us; skew 500 ppm
I20260620 03:05:32.006039 1480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:32 dist-test-slave-xv68 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:32.013742 1480 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:05:32.015069 1480 webserver.cc:460] Webserver started at http://127.0.97.62:32885/ using document root <none> and password file <none>
I20260620 03:05:32.015823 1480 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:32.015908 1480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:32.016167 1480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:32.018904 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "0b4a1d56797f48de9a6b7142bd45d1d4"
format_stamp: "Formatted at 2026-06-20 03:05:32 on dist-test-slave-xv68"
server_key: "5283efc5a5eb8312ca1f886606d43ef1"
server_key_iv: "333fbc0c7085cd186de5510dda6d048f"
server_key_version: "encryptionkey@0"
I20260620 03:05:32.019680 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "0b4a1d56797f48de9a6b7142bd45d1d4"
format_stamp: "Formatted at 2026-06-20 03:05:32 on dist-test-slave-xv68"
server_key: "5283efc5a5eb8312ca1f886606d43ef1"
server_key_iv: "333fbc0c7085cd186de5510dda6d048f"
server_key_version: "encryptionkey@0"
I20260620 03:05:32.024507 1480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:05:32.027628 1495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:32.028825 1480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260620 03:05:32.028951 1480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "0b4a1d56797f48de9a6b7142bd45d1d4"
format_stamp: "Formatted at 2026-06-20 03:05:32 on dist-test-slave-xv68"
server_key: "5283efc5a5eb8312ca1f886606d43ef1"
server_key_iv: "333fbc0c7085cd186de5510dda6d048f"
server_key_version: "encryptionkey@0"
I20260620 03:05:32.029119 1480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:32.054092 1480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:32.058238 1480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:32.058449 1480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:32.079479 1480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40011
I20260620 03:05:32.079434 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40011 every 8 connection(s)
I20260620 03:05:32.080717 1480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:05:32.082298 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 1480
I20260620 03:05:32.082525 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:05:32.083331 388 external_mini_cluster.cc:1428] Setting key 78a9c5ef8fc1a938e035a24c2cfe14db
I20260620 03:05:32.085345 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 03:05:32 dist-test-slave-xv68 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:32.098577 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4: Bootstrap starting.
I20260620 03:05:32.101805 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:32.102782 1557 log.cc:824] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:32.105394 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:32.085272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.0.97.62:40011)
0620 03:05:32.086071 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:32.086082 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:32.086125 (+ 43us) server_negotiation.cc:407] Connection header received
0620 03:05:32.087724 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:32.087744 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:32.088108 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:32.088429 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:32.090193 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:32.091597 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:32.093285 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:32.093944 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:32.094086 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:32.098566 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:32.098573 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:32.098587 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:32.098641 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:32.101545 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:32.102275 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:32.102286 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:32.102292 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:32.102441 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:32.102796 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:32.102802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:32.102806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:32.103395 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:32.103722 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:32.104095 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 03:05:32.104462 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":411,"threads_started":1}
I20260620 03:05:32.105577 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4: No bootstrap required, opened a new log
I20260620 03:05:32.112648 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4a1d56797f48de9a6b7142bd45d1d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40011 } }
I20260620 03:05:32.112982 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:32.113082 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b4a1d56797f48de9a6b7142bd45d1d4, State: Initialized, Role: FOLLOWER
I20260620 03:05:32.113679 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4a1d56797f48de9a6b7142bd45d1d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40011 } }
I20260620 03:05:32.113940 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:05:32.114073 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:05:32.114262 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:32.116335 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4a1d56797f48de9a6b7142bd45d1d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40011 } }
I20260620 03:05:32.116807 1557 leader_election.cc:302] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b4a1d56797f48de9a6b7142bd45d1d4; no voters:
I20260620 03:05:32.117355 1557 leader_election.cc:288] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:05:32.117496 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:05:32.118197 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4 [term 1 LEADER]: Becoming Leader. State: Replica: 0b4a1d56797f48de9a6b7142bd45d1d4, State: Running, Role: LEADER
I20260620 03:05:32.118793 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4a1d56797f48de9a6b7142bd45d1d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40011 } }
I20260620 03:05:32.119305 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:05:32.121922 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b4a1d56797f48de9a6b7142bd45d1d4. Latest consensus state: current_term: 1 leader_uuid: "0b4a1d56797f48de9a6b7142bd45d1d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4a1d56797f48de9a6b7142bd45d1d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40011 } } }
I20260620 03:05:32.122284 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4 [sys.catalog]: This master's current role is: LEADER
I20260620 03:05:32.122768 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b4a1d56797f48de9a6b7142bd45d1d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4a1d56797f48de9a6b7142bd45d1d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40011 } } }
I20260620 03:05:32.123059 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4 [sys.catalog]: This master's current role is: LEADER
I20260620 03:05:32.124004 1571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:05:32.129458 1571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:05:32.137622 1571 catalog_manager.cc:1347] Generated new cluster ID: ab5d3119cd504ea9ab6752974ba3387a
I20260620 03:05:32.137724 1571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:05:32.178979 1571 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:05:32.181010 1571 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:05:32.193148 1571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4: Generated new TSK 0
I20260620 03:05:32.194202 1571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:05:32.248870 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40011
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:32.562947 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:32.563222 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:32.563409 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:32.570333 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:32.570436 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:32.570617 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 03:05:32.570744 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:05:32.580470 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:32.581765 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:32.583207 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:32.596452 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:32.596494 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:32.596463 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:32.597565 1583 server_base.cc:1034] running on GCE node
I20260620 03:05:32.598111 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:32.598954 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:32.600169 1583 hybrid_clock.cc:648] HybridClock initialized: now 1781924732600112 us; error 41 us; skew 500 ppm
I20260620 03:05:32.600479 1583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:32 dist-test-slave-xv68 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:32.603704 1583 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:05:32.605152 1583 webserver.cc:460] Webserver started at http://127.0.97.1:45479/ using document root <none> and password file <none>
I20260620 03:05:32.605829 1583 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:32.605954 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:32.606273 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:32.609064 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "6aac2ab66b5c499ea24c96f74bcff0e9"
format_stamp: "Formatted at 2026-06-20 03:05:32 on dist-test-slave-xv68"
server_key: "312668e1753deb7658695ae862850e97"
server_key_iv: "7608d6a7a745ea968fa76e3f447650f7"
server_key_version: "encryptionkey@0"
I20260620 03:05:32.609834 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6aac2ab66b5c499ea24c96f74bcff0e9"
format_stamp: "Formatted at 2026-06-20 03:05:32 on dist-test-slave-xv68"
server_key: "312668e1753deb7658695ae862850e97"
server_key_iv: "7608d6a7a745ea968fa76e3f447650f7"
server_key_version: "encryptionkey@0"
I20260620 03:05:32.614660 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 03:05:32.617960 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:32.619333 1583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:05:32.619496 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "6aac2ab66b5c499ea24c96f74bcff0e9"
format_stamp: "Formatted at 2026-06-20 03:05:32 on dist-test-slave-xv68"
server_key: "312668e1753deb7658695ae862850e97"
server_key_iv: "7608d6a7a745ea968fa76e3f447650f7"
server_key_version: "encryptionkey@0"
I20260620 03:05:32.619707 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:32.634436 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:32.638471 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:32.638734 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:32.641280 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:32.641474 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 03:05:32.641621 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:32.641700 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 03:05:32 dist-test-slave-xv68 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:32.664192 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:32.641554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47433 (local address 127.0.97.62:40011)
0620 03:05:32.644122 (+ 2568us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:32.644126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:32.644143 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:05:32.646279 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:32.646285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:32.646370 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:32.647077 (+ 707us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:32.648723 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:32.649967 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:32.652299 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:32.652920 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:32.653052 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:32.656980 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:32.656989 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:32.656996 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:32.657049 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:32.660010 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:32.661084 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:32.661092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:32.661097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:32.661238 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:32.661814 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:32.661824 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:32.661830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:32.662274 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:32.662457 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:32.663392 (+ 935us) server_negotiation.cc:299] Negotiation successful
0620 03:05:32.663699 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2468,"thread_start_us":2381,"threads_started":1}
I20260620 03:05:32.665381 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:32.642212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40011 (local address 127.0.97.1:47433)
0620 03:05:32.643205 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:32.643252 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:32.645945 (+ 2693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:32.646813 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:32.646828 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:32.647690 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:32.648494 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:32.648536 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:32.650159 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:32.650167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:32.652151 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:32.652163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:32.653151 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:32.653158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:32.653359 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:32.654061 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:32.654098 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:32.656747 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:32.660146 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:32.660157 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:32.660176 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:32.660901 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:32.661333 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:32.661341 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:32.661345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:32.661641 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:32.662415 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:32.662426 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:32.662864 (+ 438us) client_negotiation.cc:769] Sending connection context
0620 03:05:32.663222 (+ 358us) client_negotiation.cc:240] Negotiation successful
0620 03:05:32.663899 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":411,"threads_started":1}
I20260620 03:05:32.684340 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35941
I20260620 03:05:32.684446 1718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35941 every 8 connection(s)
I20260620 03:05:32.685688 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:05:32.687929 1719 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40011
I20260620 03:05:32.688601 1719 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:32.689294 1719 heartbeater.cc:511] Master 127.0.97.62:40011 requested a full tablet report, sending...
I20260620 03:05:32.689448 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:32.673636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48389 (local address 127.0.97.62:40011)
0620 03:05:32.673895 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:32.673904 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:32.674608 (+ 704us) server_negotiation.cc:407] Connection header received
0620 03:05:32.674865 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:32.674874 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:32.674975 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:32.675092 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:32.676351 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:32.677590 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:32.679963 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:32.680559 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:32.680719 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:32.683065 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:32.683071 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:32.683076 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:32.683118 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:32.686257 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:32.687214 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:32.687220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:32.687223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:32.687441 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:32.687899 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:32.687906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:32.687910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:32.688221 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:32.688418 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:32.688855 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 03:05:32.689080 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 03:05:32.689606 1689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:32.673549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40011 (local address 127.0.97.1:48389)
0620 03:05:32.674469 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:32.674487 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:32.674650 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:32.675141 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:32.675148 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:32.675889 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:32.676202 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:32.676211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:32.677760 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:32.677768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:32.679734 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:32.679748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:32.680783 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:32.680793 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:32.681011 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:32.681800 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:32.681829 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:32.682894 (+ 1065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:32.686410 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:32.686422 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:32.686429 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:32.687085 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:32.687580 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:32.687585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:32.687588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:32.687737 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:32.688384 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:32.688391 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:32.688577 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 03:05:32.688754 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 03:05:32.689095 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":596,"threads_started":1}
I20260620 03:05:32.691913 1511 ts_manager.cc:194] Registered new tserver with Master: 6aac2ab66b5c499ea24c96f74bcff0e9 (127.0.97.1:35941)
I20260620 03:05:32.693689 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47433
I20260620 03:05:32.695406 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 1583
I20260620 03:05:32.695617 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:05:32.696245 388 external_mini_cluster.cc:1428] Setting key 1b0c42cb5f17c15c724370c248af24bd
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:05:32.735214 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40011
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:33.049890 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:33.050195 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:33.050361 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:33.057366 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:33.057507 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:33.057659 1723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 03:05:33.057789 1723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:05:33.067209 1723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:33.068595 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:33.069876 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:33.082075 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:33.082044 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:33.082835 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:33.083400 1723 server_base.cc:1034] running on GCE node
I20260620 03:05:33.084048 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:33.084990 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:33.086246 1723 hybrid_clock.cc:648] HybridClock initialized: now 1781924733086173 us; error 46 us; skew 500 ppm
I20260620 03:05:33.086546 1723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:33 dist-test-slave-xv68 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:33.089738 1723 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:05:33.091523 1723 webserver.cc:460] Webserver started at http://127.0.97.2:40329/ using document root <none> and password file <none>
I20260620 03:05:33.092249 1723 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:33.092373 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:33.092694 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:33.095608 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "8ddb3250842946ee85d4475ea7f77d0a"
format_stamp: "Formatted at 2026-06-20 03:05:33 on dist-test-slave-xv68"
server_key: "0d4bb6f923f100785929e94a3e388ac0"
server_key_iv: "aae92cd131af0be105443553df4cd25e"
server_key_version: "encryptionkey@0"
I20260620 03:05:33.096410 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8ddb3250842946ee85d4475ea7f77d0a"
format_stamp: "Formatted at 2026-06-20 03:05:33 on dist-test-slave-xv68"
server_key: "0d4bb6f923f100785929e94a3e388ac0"
server_key_iv: "aae92cd131af0be105443553df4cd25e"
server_key_version: "encryptionkey@0"
I20260620 03:05:33.101550 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:05:33.104918 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:33.106163 1723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 03:05:33.106343 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "8ddb3250842946ee85d4475ea7f77d0a"
format_stamp: "Formatted at 2026-06-20 03:05:33 on dist-test-slave-xv68"
server_key: "0d4bb6f923f100785929e94a3e388ac0"
server_key_iv: "aae92cd131af0be105443553df4cd25e"
server_key_version: "encryptionkey@0"
I20260620 03:05:33.106552 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:33.126232 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:33.130239 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:33.130611 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:33.133790 1723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:33.133901 1723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:33.134053 1723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:33.134130 1723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 03:05:33 dist-test-slave-xv68 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:33.155879 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:33.134373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49475 (local address 127.0.97.62:40011)
0620 03:05:33.134551 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:33.134557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:33.136042 (+ 1485us) server_negotiation.cc:407] Connection header received
0620 03:05:33.138061 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:33.138070 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:33.138184 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:33.138305 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:33.140689 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.141727 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.144518 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.144935 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.145071 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:33.149039 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:33.149044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:33.149047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:33.149079 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:33.151984 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:33.152916 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:33.152924 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:33.152931 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:33.153154 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:33.153655 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:33.153663 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:33.153669 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:33.154145 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:33.154360 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:33.155024 (+ 664us) server_negotiation.cc:299] Negotiation successful
0620 03:05:33.155351 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 03:05:33.156451 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:33.134973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40011 (local address 127.0.97.2:49475)
0620 03:05:33.135848 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:33.135904 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:33.137785 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:33.138570 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:33.138581 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:33.139540 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:33.140494 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:33.140517 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.141906 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.141911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:33.144350 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:33.144365 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.145179 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.145188 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:33.145422 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:33.146164 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:33.146186 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:33.148804 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:33.152133 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:33.152147 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:33.152170 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:33.152754 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:33.153314 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:33.153320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:33.153329 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:33.153527 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:33.154303 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:33.154308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:33.154638 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 03:05:33.154865 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 03:05:33.155232 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":402,"threads_started":1}
I20260620 03:05:33.175354 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:33.162600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43993 (local address 127.0.97.62:40011)
0620 03:05:33.162827 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:33.162835 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:33.163445 (+ 610us) server_negotiation.cc:407] Connection header received
0620 03:05:33.163650 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:33.163656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:33.163725 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:33.163827 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:33.164661 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.165366 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.166938 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.167598 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.167722 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:33.169465 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:33.169471 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:33.169475 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:33.169515 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:33.172421 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:33.173328 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:33.173336 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:33.173341 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:33.173483 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:33.173969 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:33.173976 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:33.173981 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:33.174300 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:33.174406 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:33.174930 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 03:05:33.175075 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:05:33.175997 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:33.162628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40011 (local address 127.0.97.2:43993)
0620 03:05:33.163340 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:33.163370 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:33.163519 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:33.163845 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:33.163849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:33.164268 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:33.164529 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:33.164537 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.165496 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.165501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:33.166726 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:33.166746 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.167800 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.167807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:33.167996 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:33.168603 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:33.168619 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:33.169296 (+ 677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:33.172582 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:33.172591 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:33.172595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:33.173193 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:33.173617 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:33.173628 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:33.173635 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:33.173833 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:33.174420 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:33.174428 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:33.174682 (+ 254us) client_negotiation.cc:769] Sending connection context
0620 03:05:33.174871 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 03:05:33.175365 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":371,"threads_started":1}
I20260620 03:05:33.193609 1723 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38299
I20260620 03:05:33.193639 1857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38299 every 8 connection(s)
I20260620 03:05:33.194841 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:05:33.196925 1858 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40011
I20260620 03:05:33.197109 1858 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:33.197690 1858 heartbeater.cc:511] Master 127.0.97.62:40011 requested a full tablet report, sending...
I20260620 03:05:33.198920 1511 ts_manager.cc:194] Registered new tserver with Master: 8ddb3250842946ee85d4475ea7f77d0a (127.0.97.2:38299)
I20260620 03:05:33.199714 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49475
I20260620 03:05:33.202253 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 1723
I20260620 03:05:33.202409 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:05:33.202780 388 external_mini_cluster.cc:1428] Setting key 27619cd309db2a527303c3601412a0ea
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:05:33.242446 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40011
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:33.558622 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:33.558880 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:33.558974 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:33.565874 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:33.565971 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:33.566051 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 03:05:33.566129 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:05:33.575493 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:33.576642 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:33.577857 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:33.590785 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:33.590911 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:33.590806 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:33.596808 1862 server_base.cc:1034] running on GCE node
I20260620 03:05:33.597441 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:33.598384 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:33.599622 1862 hybrid_clock.cc:648] HybridClock initialized: now 1781924733599569 us; error 38 us; skew 500 ppm
I20260620 03:05:33.599920 1862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:33 dist-test-slave-xv68 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:33.602880 1862 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:05:33.604336 1862 webserver.cc:460] Webserver started at http://127.0.97.3:46657/ using document root <none> and password file <none>
I20260620 03:05:33.605029 1862 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:33.605163 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:33.605484 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:33.608341 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1"
format_stamp: "Formatted at 2026-06-20 03:05:33 on dist-test-slave-xv68"
server_key: "902c0269f4c2d8ad5b2203329cb08e32"
server_key_iv: "0ded4700fc36e970d732a3cfce8cfcf7"
server_key_version: "encryptionkey@0"
I20260620 03:05:33.609104 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1"
format_stamp: "Formatted at 2026-06-20 03:05:33 on dist-test-slave-xv68"
server_key: "902c0269f4c2d8ad5b2203329cb08e32"
server_key_iv: "0ded4700fc36e970d732a3cfce8cfcf7"
server_key_version: "encryptionkey@0"
I20260620 03:05:33.614143 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:05:33.617421 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:33.618590 1862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 03:05:33.618763 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1"
format_stamp: "Formatted at 2026-06-20 03:05:33 on dist-test-slave-xv68"
server_key: "902c0269f4c2d8ad5b2203329cb08e32"
server_key_iv: "0ded4700fc36e970d732a3cfce8cfcf7"
server_key_version: "encryptionkey@0"
I20260620 03:05:33.618983 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:33.631902 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:33.635855 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:33.636134 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:33.638700 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:33.638805 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:33.638962 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:33.639092 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 03:05:33 dist-test-slave-xv68 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:33.661381 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:33.639170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40227 (local address 127.0.97.62:40011)
0620 03:05:33.640165 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:33.640172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:33.640716 (+ 544us) server_negotiation.cc:407] Connection header received
0620 03:05:33.642821 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:33.642826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:33.642908 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:33.643024 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:33.645166 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.646275 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.648806 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.649506 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.649663 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:33.654781 (+ 5118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:33.654792 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:33.654800 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:33.654852 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:33.658309 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:33.659100 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:33.659110 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:33.659112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:33.659233 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:33.659649 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:33.659653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:33.659656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:33.660002 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:33.660144 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:33.660823 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 03:05:33.661024 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848}
I20260620 03:05:33.662396 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:33.639717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40011 (local address 127.0.97.3:40227)
0620 03:05:33.640524 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:33.640571 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:33.642561 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:33.643285 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:33.643329 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:33.644076 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:33.644955 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:33.644984 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.646424 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.646430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:33.648646 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:33.648666 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.649993 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.650001 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:33.650389 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:33.651263 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:33.651328 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:33.654517 (+ 3189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:33.658438 (+ 3921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:33.658446 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:33.658468 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:33.658981 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:33.659363 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:33.659368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:33.659371 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:33.659535 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:33.660124 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:33.660131 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:33.660445 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 03:05:33.660693 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 03:05:33.661151 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":384,"threads_started":1}
I20260620 03:05:33.683517 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:33.668941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40011 (local address 127.0.97.3:45953)
0620 03:05:33.669580 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:33.669603 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:33.669702 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:33.670056 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:33.670060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:33.670450 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:33.670704 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:33.670711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.672321 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.672330 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:33.673936 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:33.673950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.674994 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.675004 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:33.675322 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:33.676136 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:33.676156 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:33.677709 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:33.680998 (+ 3289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:33.681003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:33.681007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:33.681389 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:33.681756 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:33.681761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:33.681765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:33.681885 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:33.682375 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:33.682380 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:33.682553 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 03:05:33.682695 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 03:05:33.683010 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":373,"threads_started":1}
I20260620 03:05:33.684335 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:33.668905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45953 (local address 127.0.97.62:40011)
0620 03:05:33.669119 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:33.669128 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:33.669693 (+ 565us) server_negotiation.cc:407] Connection header received
0620 03:05:33.669790 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:33.669798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:33.669918 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:33.670045 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:33.670832 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.672143 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.674169 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.674776 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.674932 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:33.677936 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:33.677963 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:33.677970 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:33.678009 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:33.680859 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:33.681505 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:33.681512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:33.681516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:33.681647 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:33.681987 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:33.681994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:33.681997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:33.682264 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:33.682408 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:33.682802 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 03:05:33.683009 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 03:05:33.695008 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37009
I20260620 03:05:33.695087 1996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37009 every 8 connection(s)
I20260620 03:05:33.696231 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:05:33.696310 1719 heartbeater.cc:503] Master 127.0.97.62:40011 was elected leader, sending a full tablet report...
I20260620 03:05:33.698490 1997 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40011
I20260620 03:05:33.698657 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 1862
I20260620 03:05:33.698746 1997 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:33.698855 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:05:33.699337 388 external_mini_cluster.cc:1428] Setting key ba062843dee8f28771082918b69aa418
I20260620 03:05:33.699519 1997 heartbeater.cc:511] Master 127.0.97.62:40011 requested a full tablet report, sending...
I20260620 03:05:33.700889 1511 ts_manager.cc:194] Registered new tserver with Master: e47bd8c4fffc4b3f9eef8f5f920aacb1 (127.0.97.3:37009)
I20260620 03:05:33.701757 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40227
I20260620 03:05:33.702138 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 03:05:33 dist-test-slave-xv68 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 03:05:33 dist-test-slave-xv68 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:33.747385 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:33.733020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.0.97.62:40011)
0620 03:05:33.733244 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:33.733251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:33.733557 (+ 306us) server_negotiation.cc:407] Connection header received
0620 03:05:33.733769 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:33.733774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:33.733834 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:33.733936 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:33.735317 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.736117 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.738041 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.738670 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.738844 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:33.742433 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:33.742438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:33.742441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:33.742472 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:33.745041 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:33.745606 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:33.745610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:33.745612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:33.745719 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:33.746111 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:33.746114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:33.746116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:33.746342 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:33.746515 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:33.746752 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 03:05:33.746924 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:05:33.751366 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:05:33.757983 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:05:33.780808 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:33.770898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57672 (local address 127.0.97.2:38299)
0620 03:05:33.772841 (+ 1943us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:33.772847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:33.772870 (+ 23us) server_negotiation.cc:407] Connection header received
0620 03:05:33.772948 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:33.772953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:33.773132 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:33.773287 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:33.774218 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.775787 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.778290 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.779581 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.779736 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:33.779950 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:33.780300 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 03:05:33.780474 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1809,"thread_start_us":1753,"threads_started":1}
I20260620 03:05:33.780896 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:33.769985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38299 (local address 127.0.0.1:57672)
0620 03:05:33.770898 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:33.770945 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:33.771156 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:33.773345 (+ 2189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:33.773356 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:33.773401 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:33.773973 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:33.773988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.775923 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.775928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:33.778135 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:33.778146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.779789 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.779795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:33.780013 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:33.780067 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 03:05:33.780208 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 03:05:33.780415 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":29,"thread_start_us":1564,"threads_started":1}
I20260620 03:05:33.780964 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:33.769447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35941 (local address 127.0.0.1:42554)
0620 03:05:33.773520 (+ 4073us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:33.773534 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:33.773698 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:33.774318 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:33.774322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:33.774371 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:33.774743 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:33.774755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.776386 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.776394 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:33.778131 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:33.778145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.780156 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.780165 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:33.780326 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:33.780356 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 03:05:33.780442 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 03:05:33.780577 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":4,"spinlock_wait_cycles":17792,"thread_start_us":818,"threads_started":1}
I20260620 03:05:33.781778 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:33.769702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42554 (local address 127.0.97.1:35941)
0620 03:05:33.770247 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:33.770257 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:33.773642 (+ 3385us) server_negotiation.cc:407] Connection header received
0620 03:05:33.773857 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:33.773867 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:33.774131 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:33.774322 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:33.774918 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.776259 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.778333 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.779864 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.779963 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:33.780180 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:33.781307 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0620 03:05:33.781502 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":260,"threads_started":1}
I20260620 03:05:33.782490 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:33.770396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37009 (local address 127.0.0.1:37472)
0620 03:05:33.774242 (+ 3846us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:33.774259 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:33.774367 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:33.775243 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:33.775247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:33.775266 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:33.775683 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:33.775695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.777699 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.777704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:33.779845 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:33.779858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.781694 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.781700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:33.781869 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:33.781911 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 03:05:33.781992 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 03:05:33.782111 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3736,"mutex_wait_us":50,"spinlock_wait_cycles":2304,"thread_start_us":3511,"threads_started":1}
I20260620 03:05:33.783586 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:33.771255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37472 (local address 127.0.97.3:37009)
0620 03:05:33.772132 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:33.772145 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:33.774491 (+ 2346us) server_negotiation.cc:407] Connection header received
0620 03:05:33.774579 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:33.774584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:33.774797 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:33.774969 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:33.775938 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.777536 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.780028 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:33.781498 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:33.781647 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:33.781873 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:33.782062 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 03:05:33.782219 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":996,"threads_started":1}
I20260620 03:05:33.785235 1788 tablet_service.cc:1467] Processing CreateTablet for tablet e0d4d2b36fd04d78a369e83d63b7790b (DEFAULT_TABLE table=test-table [id=5a045df1506648198c3cec42a33d0f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:33.785235 1649 tablet_service.cc:1467] Processing CreateTablet for tablet e0d4d2b36fd04d78a369e83d63b7790b (DEFAULT_TABLE table=test-table [id=5a045df1506648198c3cec42a33d0f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:33.786644 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d4d2b36fd04d78a369e83d63b7790b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:33.786659 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d4d2b36fd04d78a369e83d63b7790b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:33.786530 1927 tablet_service.cc:1467] Processing CreateTablet for tablet e0d4d2b36fd04d78a369e83d63b7790b (DEFAULT_TABLE table=test-table [id=5a045df1506648198c3cec42a33d0f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:33.787889 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d4d2b36fd04d78a369e83d63b7790b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:33.798403 2016 tablet_bootstrap.cc:492] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1: Bootstrap starting.
I20260620 03:05:33.800923 2017 tablet_bootstrap.cc:492] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a: Bootstrap starting.
I20260620 03:05:33.802466 2018 tablet_bootstrap.cc:492] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9: Bootstrap starting.
I20260620 03:05:33.803138 2016 tablet_bootstrap.cc:654] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:33.804304 2017 tablet_bootstrap.cc:654] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:33.804508 2016 log.cc:824] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:33.805459 2017 log.cc:824] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:33.805960 2018 tablet_bootstrap.cc:654] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:33.807061 2018 log.cc:824] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:33.807489 2016 tablet_bootstrap.cc:492] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1: No bootstrap required, opened a new log
I20260620 03:05:33.807816 2016 ts_tablet_manager.cc:1397] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 03:05:33.808344 2017 tablet_bootstrap.cc:492] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a: No bootstrap required, opened a new log
I20260620 03:05:33.808593 2017 ts_tablet_manager.cc:1397] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 03:05:33.809854 2018 tablet_bootstrap.cc:492] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9: No bootstrap required, opened a new log
I20260620 03:05:33.810154 2018 ts_tablet_manager.cc:1397] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 03:05:33.817351 2018 raft_consensus.cc:348] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:33.817775 2018 raft_consensus.cc:374] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:33.817929 2018 raft_consensus.cc:729] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aac2ab66b5c499ea24c96f74bcff0e9, State: Initialized, Role: FOLLOWER
I20260620 03:05:33.818539 2018 consensus_queue.cc:260] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:33.819504 2017 raft_consensus.cc:348] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:33.819998 2017 raft_consensus.cc:374] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:33.820137 2017 raft_consensus.cc:729] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ddb3250842946ee85d4475ea7f77d0a, State: Initialized, Role: FOLLOWER
I20260620 03:05:33.820518 2018 ts_tablet_manager.cc:1428] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260620 03:05:33.820715 2017 consensus_queue.cc:260] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:33.822304 2017 ts_tablet_manager.cc:1428] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260620 03:05:33.822757 1858 heartbeater.cc:503] Master 127.0.97.62:40011 was elected leader, sending a full tablet report...
I20260620 03:05:33.819495 2016 raft_consensus.cc:348] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:33.824056 2016 raft_consensus.cc:374] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:33.824167 2016 raft_consensus.cc:729] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47bd8c4fffc4b3f9eef8f5f920aacb1, State: Initialized, Role: FOLLOWER
I20260620 03:05:33.824872 2016 consensus_queue.cc:260] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:33.826124 1997 heartbeater.cc:503] Master 127.0.97.62:40011 was elected leader, sending a full tablet report...
I20260620 03:05:33.826635 2016 ts_tablet_manager.cc:1428] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1: Time spent starting tablet: real 0.019s user 0.013s sys 0.000s
W20260620 03:05:33.938169 1720 tablet.cc:2369] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:05:33.946851 1859 tablet.cc:2369] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:05:33.947912 1998 tablet.cc:2369] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:05:34.195343 2022 raft_consensus.cc:482] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:05:34.195612 2022 raft_consensus.cc:504] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.197528 2022 leader_election.cc:288] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ddb3250842946ee85d4475ea7f77d0a (127.0.97.2:38299), e47bd8c4fffc4b3f9eef8f5f920aacb1 (127.0.97.3:37009)
I20260620 03:05:34.206849 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:34.198933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37009 (local address 127.0.97.1:47151)
0620 03:05:34.199526 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:34.199549 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:34.199703 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:34.200061 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:34.200065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:34.200087 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:34.201278 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:34.201287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.202220 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.202225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:34.204770 (+ 2545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:34.204783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.206281 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.206286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:34.206390 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:34.206414 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 03:05:34.206490 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 03:05:34.206575 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":143,"thread_start_us":348,"threads_started":1}
I20260620 03:05:34.206918 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:34.197924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47151 (local address 127.0.97.3:37009)
0620 03:05:34.198179 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:34.198189 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:34.199636 (+ 1447us) server_negotiation.cc:407] Connection header received
0620 03:05:34.199837 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:34.199842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:34.199912 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:34.200013 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:34.201426 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.202067 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.204932 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.206101 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.206226 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:34.206345 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:34.206536 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 03:05:34.206658 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:05:34.208096 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:34.197869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38299 (local address 127.0.97.1:60425)
0620 03:05:34.198980 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:34.198999 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:34.199194 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:34.199793 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:34.199800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:34.199829 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:34.201049 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:34.201062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.202495 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.202500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:34.204772 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:34.204783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.207484 (+ 2701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.207490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:34.207666 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:34.207690 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 03:05:34.207764 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 03:05:34.207847 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"mutex_wait_us":137,"thread_start_us":501,"threads_started":1}
I20260620 03:05:34.208042 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d4d2b36fd04d78a369e83d63b7790b" candidate_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" is_pre_election: true
I20260620 03:05:34.208207 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:34.198228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60425 (local address 127.0.97.2:38299)
0620 03:05:34.198408 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:34.198414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:34.199105 (+ 691us) server_negotiation.cc:407] Connection header received
0620 03:05:34.199423 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:34.199431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:34.199510 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:34.199586 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:34.201231 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.202364 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.206095 (+ 3731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.207256 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.207399 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:34.207470 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:34.207839 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 03:05:34.207970 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:05:34.208510 1952 raft_consensus.cc:2454] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aac2ab66b5c499ea24c96f74bcff0e9 in term 0.
I20260620 03:05:34.209096 1602 leader_election.cc:302] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aac2ab66b5c499ea24c96f74bcff0e9, e47bd8c4fffc4b3f9eef8f5f920aacb1; no voters:
I20260620 03:05:34.209106 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d4d2b36fd04d78a369e83d63b7790b" candidate_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddb3250842946ee85d4475ea7f77d0a" is_pre_election: true
I20260620 03:05:34.209499 2022 raft_consensus.cc:2781] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:05:34.209616 2022 raft_consensus.cc:482] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:05:34.209616 1813 raft_consensus.cc:2454] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aac2ab66b5c499ea24c96f74bcff0e9 in term 0.
I20260620 03:05:34.209775 2022 raft_consensus.cc:3037] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:34.212531 2022 raft_consensus.cc:504] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.213141 2022 leader_election.cc:288] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [CANDIDATE]: Term 1 election: Requested vote from peers 8ddb3250842946ee85d4475ea7f77d0a (127.0.97.2:38299), e47bd8c4fffc4b3f9eef8f5f920aacb1 (127.0.97.3:37009)
I20260620 03:05:34.213459 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d4d2b36fd04d78a369e83d63b7790b" candidate_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddb3250842946ee85d4475ea7f77d0a"
I20260620 03:05:34.213522 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d4d2b36fd04d78a369e83d63b7790b" candidate_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1"
I20260620 03:05:34.213644 1813 raft_consensus.cc:3037] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:34.213779 1952 raft_consensus.cc:3037] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:34.215739 1813 raft_consensus.cc:2454] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aac2ab66b5c499ea24c96f74bcff0e9 in term 1.
I20260620 03:05:34.215891 1952 raft_consensus.cc:2454] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aac2ab66b5c499ea24c96f74bcff0e9 in term 1.
I20260620 03:05:34.216140 1600 leader_election.cc:302] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aac2ab66b5c499ea24c96f74bcff0e9, 8ddb3250842946ee85d4475ea7f77d0a; no voters:
I20260620 03:05:34.216486 2022 raft_consensus.cc:2781] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:05:34.216898 2022 raft_consensus.cc:686] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 1 LEADER]: Becoming Leader. State: Replica: 6aac2ab66b5c499ea24c96f74bcff0e9, State: Running, Role: LEADER
I20260620 03:05:34.217343 2022 consensus_queue.cc:237] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.221868 1511 catalog_manager.cc:5495] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6aac2ab66b5c499ea24c96f74bcff0e9 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } health_report { overall_health: UNKNOWN } } }
I20260620 03:05:34.351187 2030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:34.345364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.0.97.1:35941)
0620 03:05:34.345769 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:34.345775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:34.346241 (+ 466us) server_negotiation.cc:407] Connection header received
0620 03:05:34.346481 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:34.346485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:34.346556 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:34.346683 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:05:34.347097 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.347818 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.349237 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.349613 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.349702 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:34.350164 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:05:34.350643 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:05:34.350748 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:34.350910 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 03:05:34.350970 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":226,"threads_started":1}
I20260620 03:05:34.358870 1813 raft_consensus.cc:1261] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a [term 1 FOLLOWER]: Refusing update from remote peer 6aac2ab66b5c499ea24c96f74bcff0e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:05:34.358888 1952 raft_consensus.cc:1261] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 1 FOLLOWER]: Refusing update from remote peer 6aac2ab66b5c499ea24c96f74bcff0e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:05:34.359817 2027 consensus_queue.cc:1035] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:05:34.360220 2022 consensus_queue.cc:1035] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:05:34.372928 2032 mvcc.cc:205] Tried to move back new op lower bound from 7298763711924199424 to 7298763711357624320. Current Snapshot: MvccSnapshot[applied={T|T < 7298763711924199424}]
I20260620 03:05:34.373958 2036 mvcc.cc:205] Tried to move back new op lower bound from 7298763711924199424 to 7298763711357624320. Current Snapshot: MvccSnapshot[applied={T|T < 7298763711924199424}]
I20260620 03:05:34.391362 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:05:34.391638 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:05:34.394852 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4: Sending DeleteTablet for 3 replicas of tablet e0d4d2b36fd04d78a369e83d63b7790b
I20260620 03:05:34.395905 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet e0d4d2b36fd04d78a369e83d63b7790b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:05:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42554
I20260620 03:05:34.396054 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet e0d4d2b36fd04d78a369e83d63b7790b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:05:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37472
I20260620 03:05:34.396688 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet e0d4d2b36fd04d78a369e83d63b7790b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:05:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57672
I20260620 03:05:34.397848 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:05:34.398658 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:05:34.408594 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 66521bbef8564b5abe14340eac7981d6 (DEFAULT_TABLE table=test-table [id=a95bccc2adaf4c0587d0238f5eaf45fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:34.408882 2044 tablet_replica.cc:330] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1: stopping tablet replica
I20260620 03:05:34.409065 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 66521bbef8564b5abe14340eac7981d6 (DEFAULT_TABLE table=test-table [id=a95bccc2adaf4c0587d0238f5eaf45fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:34.409189 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66521bbef8564b5abe14340eac7981d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:34.409328 2044 raft_consensus.cc:2229] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:05:34.409520 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66521bbef8564b5abe14340eac7981d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:34.409833 2044 raft_consensus.cc:2258] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:05:34.410704 2045 tablet_replica.cc:330] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a: stopping tablet replica
I20260620 03:05:34.411093 2045 raft_consensus.cc:2229] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:05:34.411599 2045 raft_consensus.cc:2258] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:05:34.415690 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 66521bbef8564b5abe14340eac7981d6 (DEFAULT_TABLE table=test-table [id=a95bccc2adaf4c0587d0238f5eaf45fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:34.416083 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66521bbef8564b5abe14340eac7981d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:34.416847 2045 ts_tablet_manager.cc:1905] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:05:34.419916 2044 ts_tablet_manager.cc:1905] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:05:34.421392 2046 tablet_bootstrap.cc:492] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a: Bootstrap starting.
I20260620 03:05:34.425166 2046 tablet_bootstrap.cc:654] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:34.427569 2043 tablet_replica.cc:330] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9: stopping tablet replica
I20260620 03:05:34.428084 2043 raft_consensus.cc:2229] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:05:34.428846 2043 raft_consensus.cc:2258] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:05:34.429374 2047 tablet_bootstrap.cc:492] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9: Bootstrap starting.
I20260620 03:05:34.430796 2045 ts_tablet_manager.cc:1918] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:05:34.432547 2048 tablet_bootstrap.cc:492] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1: Bootstrap starting.
I20260620 03:05:34.434741 2048 tablet_bootstrap.cc:654] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:34.432021 2043 ts_tablet_manager.cc:1905] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:05:34.435599 2044 ts_tablet_manager.cc:1918] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:05:34.435837 2045 log.cc:1196] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/e0d4d2b36fd04d78a369e83d63b7790b
I20260620 03:05:34.436508 2045 ts_tablet_manager.cc:1939] T e0d4d2b36fd04d78a369e83d63b7790b P 8ddb3250842946ee85d4475ea7f77d0a: Deleting consensus metadata
I20260620 03:05:34.437884 2047 tablet_bootstrap.cc:654] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:34.438340 1497 catalog_manager.cc:4841] TS 8ddb3250842946ee85d4475ea7f77d0a (127.0.97.2:38299): tablet e0d4d2b36fd04d78a369e83d63b7790b (table test-table [id=5a045df1506648198c3cec42a33d0f1b]) successfully deleted
I20260620 03:05:34.438937 2044 log.cc:1196] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/e0d4d2b36fd04d78a369e83d63b7790b
I20260620 03:05:34.439636 2044 ts_tablet_manager.cc:1939] T e0d4d2b36fd04d78a369e83d63b7790b P e47bd8c4fffc4b3f9eef8f5f920aacb1: Deleting consensus metadata
I20260620 03:05:34.441277 1499 catalog_manager.cc:4841] TS e47bd8c4fffc4b3f9eef8f5f920aacb1 (127.0.97.3:37009): tablet e0d4d2b36fd04d78a369e83d63b7790b (table test-table [id=5a045df1506648198c3cec42a33d0f1b]) successfully deleted
I20260620 03:05:34.443696 2048 tablet_bootstrap.cc:492] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1: No bootstrap required, opened a new log
I20260620 03:05:34.443913 2048 ts_tablet_manager.cc:1397] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.010s
I20260620 03:05:34.444689 2048 raft_consensus.cc:348] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.445010 2048 raft_consensus.cc:374] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:34.445165 2048 raft_consensus.cc:729] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47bd8c4fffc4b3f9eef8f5f920aacb1, State: Initialized, Role: FOLLOWER
I20260620 03:05:34.445472 2046 tablet_bootstrap.cc:492] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a: No bootstrap required, opened a new log
I20260620 03:05:34.445472 2048 consensus_queue.cc:260] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.445650 2046 ts_tablet_manager.cc:1397] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.000s
I20260620 03:05:34.446072 2048 ts_tablet_manager.cc:1428] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 03:05:34.446475 2046 raft_consensus.cc:348] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.446795 2046 raft_consensus.cc:374] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:34.447796 2046 raft_consensus.cc:729] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ddb3250842946ee85d4475ea7f77d0a, State: Initialized, Role: FOLLOWER
I20260620 03:05:34.448146 2046 consensus_queue.cc:260] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.448726 2046 ts_tablet_manager.cc:1428] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 03:05:34.450421 2047 tablet_bootstrap.cc:492] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9: No bootstrap required, opened a new log
I20260620 03:05:34.450634 2047 ts_tablet_manager.cc:1397] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.008s
I20260620 03:05:34.452080 2047 raft_consensus.cc:348] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.452826 2047 raft_consensus.cc:374] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:34.452877 2043 ts_tablet_manager.cc:1918] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:05:34.452963 2047 raft_consensus.cc:729] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aac2ab66b5c499ea24c96f74bcff0e9, State: Initialized, Role: FOLLOWER
I20260620 03:05:34.453104 2043 log.cc:1196] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/e0d4d2b36fd04d78a369e83d63b7790b
I20260620 03:05:34.453238 2047 consensus_queue.cc:260] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.453966 2043 ts_tablet_manager.cc:1939] T e0d4d2b36fd04d78a369e83d63b7790b P 6aac2ab66b5c499ea24c96f74bcff0e9: Deleting consensus metadata
I20260620 03:05:34.453958 2047 ts_tablet_manager.cc:1428] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260620 03:05:34.455792 1498 catalog_manager.cc:4841] TS 6aac2ab66b5c499ea24c96f74bcff0e9 (127.0.97.1:35941): tablet e0d4d2b36fd04d78a369e83d63b7790b (table test-table [id=5a045df1506648198c3cec42a33d0f1b]) successfully deleted
I20260620 03:05:34.551704 2033 raft_consensus.cc:482] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:05:34.551913 2033 raft_consensus.cc:504] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.554001 2033 leader_election.cc:288] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aac2ab66b5c499ea24c96f74bcff0e9 (127.0.97.1:35941), e47bd8c4fffc4b3f9eef8f5f920aacb1 (127.0.97.3:37009)
I20260620 03:05:34.562459 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:34.554950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34505 (local address 127.0.97.3:37009)
0620 03:05:34.555116 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:34.555121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:34.555572 (+ 451us) server_negotiation.cc:407] Connection header received
0620 03:05:34.555807 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:34.555813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:34.555890 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:34.555982 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:34.556658 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.557691 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.559828 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.561459 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.561593 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:34.561708 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:34.562046 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 03:05:34.562187 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 03:05:34.562781 2030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:34.554449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41999 (local address 127.0.97.1:35941)
0620 03:05:34.554655 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:34.554664 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:34.555264 (+ 600us) server_negotiation.cc:407] Connection header received
0620 03:05:34.555593 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:34.555599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:34.555677 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:34.555794 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:34.556419 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.557610 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.560206 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.561867 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.562009 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:34.562140 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:34.562396 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 03:05:34.562520 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 03:05:34.562871 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:34.554307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35941 (local address 127.0.97.2:41999)
0620 03:05:34.555139 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:34.555161 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:34.555416 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:34.555818 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:34.555822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:34.555844 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:34.556257 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:34.556267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.557762 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.557768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:34.560047 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:34.560061 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.562064 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.562070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:34.562240 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:34.562263 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 03:05:34.562368 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 03:05:34.562490 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":70,"thread_start_us":522,"threads_started":1}
I20260620 03:05:34.563225 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:34.554457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37009 (local address 127.0.97.2:34505)
0620 03:05:34.555483 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:34.555500 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:34.555664 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:34.556001 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:34.556006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:34.556036 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:34.556455 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:34.556470 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.557802 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.557808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:34.559666 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:34.559678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.561659 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.561666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:34.561845 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:34.561882 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 03:05:34.562698 (+ 816us) client_negotiation.cc:240] Negotiation successful
0620 03:05:34.562841 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":3,"thread_start_us":683,"threads_started":1}
I20260620 03:05:34.563900 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66521bbef8564b5abe14340eac7981d6" candidate_uuid: "8ddb3250842946ee85d4475ea7f77d0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" is_pre_election: true
I20260620 03:05:34.564201 1951 raft_consensus.cc:2454] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ddb3250842946ee85d4475ea7f77d0a in term 0.
I20260620 03:05:34.564116 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66521bbef8564b5abe14340eac7981d6" candidate_uuid: "8ddb3250842946ee85d4475ea7f77d0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" is_pre_election: true
I20260620 03:05:34.564606 1668 raft_consensus.cc:2454] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ddb3250842946ee85d4475ea7f77d0a in term 0.
I20260620 03:05:34.564677 1742 leader_election.cc:302] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ddb3250842946ee85d4475ea7f77d0a, e47bd8c4fffc4b3f9eef8f5f920aacb1; no voters:
I20260620 03:05:34.565063 2033 raft_consensus.cc:2781] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:05:34.565197 2033 raft_consensus.cc:482] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:05:34.565294 2033 raft_consensus.cc:3037] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:34.567624 2033 raft_consensus.cc:504] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.568233 2033 leader_election.cc:288] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [CANDIDATE]: Term 1 election: Requested vote from peers 6aac2ab66b5c499ea24c96f74bcff0e9 (127.0.97.1:35941), e47bd8c4fffc4b3f9eef8f5f920aacb1 (127.0.97.3:37009)
I20260620 03:05:34.568668 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66521bbef8564b5abe14340eac7981d6" candidate_uuid: "8ddb3250842946ee85d4475ea7f77d0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9"
I20260620 03:05:34.568872 1668 raft_consensus.cc:3037] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:34.568820 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66521bbef8564b5abe14340eac7981d6" candidate_uuid: "8ddb3250842946ee85d4475ea7f77d0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1"
I20260620 03:05:34.569020 1951 raft_consensus.cc:3037] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:34.572439 1951 raft_consensus.cc:2454] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ddb3250842946ee85d4475ea7f77d0a in term 1.
I20260620 03:05:34.572528 1668 raft_consensus.cc:2454] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ddb3250842946ee85d4475ea7f77d0a in term 1.
I20260620 03:05:34.572826 1742 leader_election.cc:302] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ddb3250842946ee85d4475ea7f77d0a, e47bd8c4fffc4b3f9eef8f5f920aacb1; no voters:
I20260620 03:05:34.573170 2033 raft_consensus.cc:2781] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:05:34.574041 2033 raft_consensus.cc:686] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [term 1 LEADER]: Becoming Leader. State: Replica: 8ddb3250842946ee85d4475ea7f77d0a, State: Running, Role: LEADER
I20260620 03:05:34.574640 2033 consensus_queue.cc:237] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.578009 1512 catalog_manager.cc:5495] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ddb3250842946ee85d4475ea7f77d0a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8ddb3250842946ee85d4475ea7f77d0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } health_report { overall_health: UNKNOWN } } }
Jun 20 03:05:34 dist-test-slave-xv68 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:34.655193 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:34.638487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40011 (local address 127.0.0.1:37820)
0620 03:05:34.639823 (+ 1336us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:34.639845 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:34.639996 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:34.640432 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:34.640438 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:34.641221 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:34.641524 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:34.641535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.642600 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.642606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:34.644584 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:34.644598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.645297 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.645304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:34.645496 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:34.646167 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:34.646189 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:34.648735 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:34.652805 (+ 4070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:34.652815 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:34.652820 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:34.653258 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:34.653680 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:34.653683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:34.653685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:34.653817 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:34.654426 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:34.654432 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:34.654572 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 03:05:34.654720 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 03:05:34.654903 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1157,"thread_start_us":990,"threads_started":1}
I20260620 03:05:34.655540 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:34.638487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37820 (local address 127.0.97.62:40011)
0620 03:05:34.640154 (+ 1667us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:34.640158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:34.640176 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:05:34.640220 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:34.640223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:34.640276 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:34.640362 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:34.641681 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.642465 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.644733 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.645132 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.645221 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:34.648941 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:34.648948 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:34.648952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:34.648986 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:34.652668 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:34.653407 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:34.653413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:34.653417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:34.653569 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:34.653940 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:34.653945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:34.653948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:34.654268 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:34.654414 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:34.654861 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 03:05:34.655058 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1576,"thread_start_us":1439,"threads_started":1}
I20260620 03:05:34.660967 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37820:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 03:05:34.662539 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:05:34.669420 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 4c69f9a488a04fc9b2cf01431e614354 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=598eb77ad9e6469a83d08fb664583a59]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 03:05:34.669405 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 4c69f9a488a04fc9b2cf01431e614354 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=598eb77ad9e6469a83d08fb664583a59]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 03:05:34.669916 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c69f9a488a04fc9b2cf01431e614354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:34.669934 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c69f9a488a04fc9b2cf01431e614354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:34.674932 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 4c69f9a488a04fc9b2cf01431e614354 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=598eb77ad9e6469a83d08fb664583a59]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 03:05:34.675458 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c69f9a488a04fc9b2cf01431e614354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:34.677157 2047 tablet_bootstrap.cc:492] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9: Bootstrap starting.
I20260620 03:05:34.677163 2046 tablet_bootstrap.cc:492] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a: Bootstrap starting.
I20260620 03:05:34.678824 2046 tablet_bootstrap.cc:654] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:34.679601 2047 tablet_bootstrap.cc:654] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:34.682989 2048 tablet_bootstrap.cc:492] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1: Bootstrap starting.
I20260620 03:05:34.684731 2048 tablet_bootstrap.cc:654] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:34.719024 2048 tablet_bootstrap.cc:492] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1: No bootstrap required, opened a new log
I20260620 03:05:34.719264 2048 ts_tablet_manager.cc:1397] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1: Time spent bootstrapping tablet: real 0.036s user 0.025s sys 0.007s
I20260620 03:05:34.720117 2048 raft_consensus.cc:348] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.720506 2048 raft_consensus.cc:374] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:34.720621 2048 raft_consensus.cc:729] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47bd8c4fffc4b3f9eef8f5f920aacb1, State: Initialized, Role: FOLLOWER
I20260620 03:05:34.720955 2047 tablet_bootstrap.cc:492] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9: No bootstrap required, opened a new log
I20260620 03:05:34.720956 2048 consensus_queue.cc:260] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.721491 2048 ts_tablet_manager.cc:1428] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 03:05:34.721383 2034 tablet_replica.cc:439] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } } }
I20260620 03:05:34.721729 2034 tablet_replica.cc:442] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:05:34.722318 2047 ts_tablet_manager.cc:1397] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9: Time spent bootstrapping tablet: real 0.044s user 0.020s sys 0.016s
I20260620 03:05:34.723161 2046 tablet_bootstrap.cc:492] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a: No bootstrap required, opened a new log
I20260620 03:05:34.723412 2046 ts_tablet_manager.cc:1397] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.011s
I20260620 03:05:34.723182 2047 raft_consensus.cc:348] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.723528 2047 raft_consensus.cc:374] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:34.723659 2047 raft_consensus.cc:729] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aac2ab66b5c499ea24c96f74bcff0e9, State: Initialized, Role: FOLLOWER
I20260620 03:05:34.723943 2047 consensus_queue.cc:260] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.724874 2022 tablet_replica.cc:439] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } } }
I20260620 03:05:34.724931 2046 raft_consensus.cc:348] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.725160 2022 tablet_replica.cc:442] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:05:34.725180 2046 raft_consensus.cc:374] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:34.725253 2046 raft_consensus.cc:729] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ddb3250842946ee85d4475ea7f77d0a, State: Initialized, Role: FOLLOWER
I20260620 03:05:34.725477 2046 consensus_queue.cc:260] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.724874 2047 ts_tablet_manager.cc:1428] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 03:05:34.726872 2046 ts_tablet_manager.cc:1428] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 03:05:34.726878 2033 tablet_replica.cc:439] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } } }
I20260620 03:05:34.727226 2033 tablet_replica.cc:442] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:05:34.827111 2033 raft_consensus.cc:482] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:05:34.827404 2033 raft_consensus.cc:504] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.828061 2033 leader_election.cc:288] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aac2ab66b5c499ea24c96f74bcff0e9 (127.0.97.1:35941), e47bd8c4fffc4b3f9eef8f5f920aacb1 (127.0.97.3:37009)
I20260620 03:05:34.828588 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c69f9a488a04fc9b2cf01431e614354" candidate_uuid: "8ddb3250842946ee85d4475ea7f77d0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" is_pre_election: true
I20260620 03:05:34.828591 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c69f9a488a04fc9b2cf01431e614354" candidate_uuid: "8ddb3250842946ee85d4475ea7f77d0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" is_pre_election: true
I20260620 03:05:34.828837 1951 raft_consensus.cc:2454] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ddb3250842946ee85d4475ea7f77d0a in term 0.
I20260620 03:05:34.828853 1668 raft_consensus.cc:2454] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ddb3250842946ee85d4475ea7f77d0a in term 0.
I20260620 03:05:34.829288 1741 leader_election.cc:302] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aac2ab66b5c499ea24c96f74bcff0e9, 8ddb3250842946ee85d4475ea7f77d0a; no voters:
I20260620 03:05:34.829622 2033 raft_consensus.cc:2781] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:05:34.829725 2033 raft_consensus.cc:482] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:05:34.829864 2033 raft_consensus.cc:3037] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:34.832057 2033 raft_consensus.cc:504] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.832643 2033 leader_election.cc:288] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [CANDIDATE]: Term 1 election: Requested vote from peers 6aac2ab66b5c499ea24c96f74bcff0e9 (127.0.97.1:35941), e47bd8c4fffc4b3f9eef8f5f920aacb1 (127.0.97.3:37009)
I20260620 03:05:34.833065 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c69f9a488a04fc9b2cf01431e614354" candidate_uuid: "8ddb3250842946ee85d4475ea7f77d0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9"
I20260620 03:05:34.833142 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c69f9a488a04fc9b2cf01431e614354" candidate_uuid: "8ddb3250842946ee85d4475ea7f77d0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1"
I20260620 03:05:34.833293 1668 raft_consensus.cc:3037] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:34.833391 1951 raft_consensus.cc:3037] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:34.835415 1668 raft_consensus.cc:2454] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ddb3250842946ee85d4475ea7f77d0a in term 1.
I20260620 03:05:34.835765 1741 leader_election.cc:302] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aac2ab66b5c499ea24c96f74bcff0e9, 8ddb3250842946ee85d4475ea7f77d0a; no voters:
I20260620 03:05:34.836020 2033 raft_consensus.cc:2781] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:05:34.836203 2033 raft_consensus.cc:686] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [term 1 LEADER]: Becoming Leader. State: Replica: 8ddb3250842946ee85d4475ea7f77d0a, State: Running, Role: LEADER
I20260620 03:05:34.836270 1951 raft_consensus.cc:2454] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ddb3250842946ee85d4475ea7f77d0a in term 1.
I20260620 03:05:34.836472 2033 consensus_queue.cc:237] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } }
I20260620 03:05:34.837980 2054 tablet_replica.cc:439] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a: TxnStatusTablet state changed. Reason: New leader 8ddb3250842946ee85d4475ea7f77d0a. Latest consensus state: current_term: 1 leader_uuid: "8ddb3250842946ee85d4475ea7f77d0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } } }
I20260620 03:05:34.838304 2054 tablet_replica.cc:442] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a: This TxnStatusTablet replica's current role is: LEADER
I20260620 03:05:34.838945 2067 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 03:05:34.839176 2067 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 03:05:34.839674 1512 catalog_manager.cc:5495] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ddb3250842946ee85d4475ea7f77d0a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8ddb3250842946ee85d4475ea7f77d0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } health_report { overall_health: UNKNOWN } } }
I20260620 03:05:34.896306 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:34.873255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57688 (local address 127.0.97.2:38299)
0620 03:05:34.888153 (+ 14898us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:34.888161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:34.888201 (+ 40us) server_negotiation.cc:407] Connection header received
0620 03:05:34.888343 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:34.888352 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:34.888499 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:34.888682 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:05:34.889258 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.891175 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.893473 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.894014 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.894162 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:34.894690 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:05:34.895512 (+ 822us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:05:34.895657 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:34.895824 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 03:05:34.895917 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14731,"thread_start_us":14559,"threads_started":1}
I20260620 03:05:34.896376 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:34.873500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38299 (local address 127.0.0.1:57688)
0620 03:05:34.873734 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:34.873755 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:34.873918 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:34.888790 (+ 14872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:34.888796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:34.888813 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 03:05:34.889030 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:34.889073 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.891414 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.891422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:34.893329 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:34.893351 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.894179 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.894185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:34.894384 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:34.894470 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 03:05:34.895625 (+ 1155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 03:05:34.895658 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 03:05:34.895772 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 03:05:34.895967 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 03:05:34.902104 1951 raft_consensus.cc:1261] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 1 FOLLOWER]: Refusing update from remote peer 8ddb3250842946ee85d4475ea7f77d0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:05:34.902645 1668 raft_consensus.cc:1261] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 1 FOLLOWER]: Refusing update from remote peer 8ddb3250842946ee85d4475ea7f77d0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:05:34.903017 2054 consensus_queue.cc:1035] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:05:34.903445 2033 consensus_queue.cc:1035] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:05:34.908645 2034 tablet_replica.cc:439] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1: TxnStatusTablet state changed. Reason: New leader 8ddb3250842946ee85d4475ea7f77d0a. Latest consensus state: current_term: 1 leader_uuid: "8ddb3250842946ee85d4475ea7f77d0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } } }
I20260620 03:05:34.908926 2034 tablet_replica.cc:442] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:05:34.913014 2054 tablet_replica.cc:439] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ddb3250842946ee85d4475ea7f77d0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } } }
I20260620 03:05:34.913281 2054 tablet_replica.cc:442] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a: This TxnStatusTablet replica's current role is: LEADER
I20260620 03:05:34.915791 2033 tablet_replica.cc:439] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ddb3250842946ee85d4475ea7f77d0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } } }
I20260620 03:05:34.916095 2033 tablet_replica.cc:442] T 4c69f9a488a04fc9b2cf01431e614354 P 8ddb3250842946ee85d4475ea7f77d0a: This TxnStatusTablet replica's current role is: LEADER
I20260620 03:05:34.911216 2022 tablet_replica.cc:439] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9: TxnStatusTablet state changed. Reason: New leader 8ddb3250842946ee85d4475ea7f77d0a. Latest consensus state: current_term: 1 leader_uuid: "8ddb3250842946ee85d4475ea7f77d0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } } }
I20260620 03:05:34.916397 2022 tablet_replica.cc:442] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:05:34.920985 2072 mvcc.cc:205] Tried to move back new op lower bound from 7298763714151493632 to 7298763713892331520. Current Snapshot: MvccSnapshot[applied={T|T < 7298763714151493632}]
I20260620 03:05:34.922523 2034 tablet_replica.cc:439] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ddb3250842946ee85d4475ea7f77d0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } } }
I20260620 03:05:34.922757 2034 tablet_replica.cc:442] T 4c69f9a488a04fc9b2cf01431e614354 P e47bd8c4fffc4b3f9eef8f5f920aacb1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:05:34.927861 2022 tablet_replica.cc:439] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ddb3250842946ee85d4475ea7f77d0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 } } peers { permanent_uuid: "8ddb3250842946ee85d4475ea7f77d0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38299 } } peers { permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 } } }
I20260620 03:05:34.928143 2022 tablet_replica.cc:442] T 4c69f9a488a04fc9b2cf01431e614354 P 6aac2ab66b5c499ea24c96f74bcff0e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:05:34.941313 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:34.932457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57702 (local address 127.0.97.2:38299)
0620 03:05:34.932631 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:34.932637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:34.935894 (+ 3257us) server_negotiation.cc:407] Connection header received
0620 03:05:34.936034 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:34.936040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:34.936135 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:34.936236 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:05:34.936657 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.937791 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.939272 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.939961 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.940092 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:34.940555 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:05:34.940687 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:05:34.940782 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:34.940923 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 03:05:34.941000 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 03:05:34.958809 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:34.952674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39499 (local address 127.0.97.2:38299)
0620 03:05:34.952840 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:34.952844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:34.953516 (+ 672us) server_negotiation.cc:407] Connection header received
0620 03:05:34.953740 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:34.953746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:34.953826 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:34.953929 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:05:34.954418 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.955216 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.957010 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.957453 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.957593 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:34.957988 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:05:34.958081 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:05:34.958171 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:34.958316 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 03:05:34.958413 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":50}
I20260620 03:05:34.958875 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:34.952792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38299 (local address 127.0.97.2:39499)
0620 03:05:34.953417 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:34.953433 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:34.953597 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:34.953933 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:34.953937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:34.953947 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 03:05:34.954284 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:34.954296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.955355 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.955363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:34.956877 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:34.956888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:34.957634 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:34.957642 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:34.957768 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:34.957820 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 03:05:34.958188 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 03:05:34.958199 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 03:05:34.958302 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 03:05:34.958450 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":3,"thread_start_us":304,"threads_started":1}
I20260620 03:05:34.967201 2033 consensus_queue.cc:1035] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47bd8c4fffc4b3f9eef8f5f920aacb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:05:34.967541 1668 raft_consensus.cc:1261] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 1 FOLLOWER]: Refusing update from remote peer 8ddb3250842946ee85d4475ea7f77d0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:05:34.968025 2033 consensus_queue.cc:1035] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aac2ab66b5c499ea24c96f74bcff0e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:05:35.048650 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:05:35.048902 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:05:35.051880 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b4a1d56797f48de9a6b7142bd45d1d4: Sending DeleteTablet for 3 replicas of tablet 66521bbef8564b5abe14340eac7981d6
I20260620 03:05:35.052603 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 66521bbef8564b5abe14340eac7981d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:05:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42554
I20260620 03:05:35.052865 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 66521bbef8564b5abe14340eac7981d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:05:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37472
I20260620 03:05:35.053020 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 66521bbef8564b5abe14340eac7981d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:05:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57672
I20260620 03:05:35.064256 2091 tablet_replica.cc:330] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a: stopping tablet replica
I20260620 03:05:35.064554 2091 raft_consensus.cc:2229] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:05:35.065253 2091 raft_consensus.cc:2258] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:05:35.067018 2090 tablet_replica.cc:330] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1: stopping tablet replica
I20260620 03:05:35.067210 2089 tablet_replica.cc:330] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9: stopping tablet replica
I20260620 03:05:35.067421 2089 raft_consensus.cc:2229] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:05:35.067660 2089 raft_consensus.cc:2258] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:05:35.067226 2090 raft_consensus.cc:2229] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:05:35.068657 2090 raft_consensus.cc:2258] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:05:35.068688 2089 ts_tablet_manager.cc:1905] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:05:35.068920 2091 ts_tablet_manager.cc:1905] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:05:35.072104 2090 ts_tablet_manager.cc:1905] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:05:35.075950 2089 ts_tablet_manager.cc:1918] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 03:05:35.076086 2089 log.cc:1196] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/66521bbef8564b5abe14340eac7981d6
I20260620 03:05:35.076525 2089 ts_tablet_manager.cc:1939] T 66521bbef8564b5abe14340eac7981d6 P 6aac2ab66b5c499ea24c96f74bcff0e9: Deleting consensus metadata
I20260620 03:05:35.077633 1498 catalog_manager.cc:4841] TS 6aac2ab66b5c499ea24c96f74bcff0e9 (127.0.97.1:35941): tablet 66521bbef8564b5abe14340eac7981d6 (table test-table [id=a95bccc2adaf4c0587d0238f5eaf45fc]) successfully deleted
I20260620 03:05:35.079130 2090 ts_tablet_manager.cc:1918] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 03:05:35.079273 2090 log.cc:1196] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/66521bbef8564b5abe14340eac7981d6
I20260620 03:05:35.079756 2090 ts_tablet_manager.cc:1939] T 66521bbef8564b5abe14340eac7981d6 P e47bd8c4fffc4b3f9eef8f5f920aacb1: Deleting consensus metadata
I20260620 03:05:35.080804 1499 catalog_manager.cc:4841] TS e47bd8c4fffc4b3f9eef8f5f920aacb1 (127.0.97.3:37009): tablet 66521bbef8564b5abe14340eac7981d6 (table test-table [id=a95bccc2adaf4c0587d0238f5eaf45fc]) successfully deleted
I20260620 03:05:35.082337 2091 ts_tablet_manager.cc:1918] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 03:05:35.082489 2091 log.cc:1196] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/66521bbef8564b5abe14340eac7981d6
I20260620 03:05:35.082899 2091 ts_tablet_manager.cc:1939] T 66521bbef8564b5abe14340eac7981d6 P 8ddb3250842946ee85d4475ea7f77d0a: Deleting consensus metadata
I20260620 03:05:35.084010 1497 catalog_manager.cc:4841] TS 8ddb3250842946ee85d4475ea7f77d0a (127.0.97.2:38299): tablet 66521bbef8564b5abe14340eac7981d6 (table test-table [id=a95bccc2adaf4c0587d0238f5eaf45fc]) successfully deleted
Jun 20 03:05:35 dist-test-slave-xv68 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:05:35.106163 2093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:35.070088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.0.97.1:35941)
0620 03:05:35.087727 (+ 17639us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:35.087734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:35.087754 (+ 20us) server_negotiation.cc:407] Connection header received
0620 03:05:35.087816 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:35.087820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:35.087897 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:35.088008 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:35.089873 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:35.091013 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:35.092945 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:35.093563 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:35.093704 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:35.099227 (+ 5523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:35.099245 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:35.099266 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:35.099337 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:35.103106 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:35.103992 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:35.104003 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:35.104007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:35.104163 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:35.104551 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:35.104557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:35.104560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:35.104979 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:35.105271 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:35.105555 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 03:05:35.105790 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17500,"thread_start_us":17374,"threads_started":1}
W20260620 03:05:35.107604 1714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42572
I20260620 03:05:35.124249 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:35.110536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37832 (local address 127.0.97.62:40011)
0620 03:05:35.110863 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:35.110871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:35.111441 (+ 570us) server_negotiation.cc:407] Connection header received
0620 03:05:35.111645 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:35.111651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:35.111754 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:35.111860 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:35.113087 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:35.113815 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:35.115157 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:35.115690 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:35.115839 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:35.118475 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:35.118484 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:35.118493 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:35.118526 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:35.121363 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:35.121972 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:35.121979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:35.121983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:35.122106 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:35.122527 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:35.122533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:35.122536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:35.122905 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:35.123145 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:35.123568 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 03:05:35.123882 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
W20260620 03:05:35.124905 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37832
I20260620 03:05:35.127429 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 1583
I20260620 03:05:35.142477 2093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:35.129572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42584 (local address 127.0.97.1:35941)
0620 03:05:35.129792 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:35.129799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:35.129819 (+ 20us) server_negotiation.cc:407] Connection header received
0620 03:05:35.129905 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:35.129910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:35.129990 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:35.130088 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:35.131188 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:35.132276 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:35.133520 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:35.134130 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:35.134267 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:35.136455 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:35.136461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:35.136466 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:35.136508 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:35.139898 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:35.140529 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:35.140535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:35.140539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:35.140695 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:35.141106 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:35.141111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:35.141114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:35.141429 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:35.141653 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:35.141938 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 03:05:35.142156 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 03:05:35.143165 1714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42584
W20260620 03:05:35.143786 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 1583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:05:35.144018 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 1583
I20260620 03:05:35.166823 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 1723
Jun 20 03:05:35 dist-test-slave-xv68 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:05:35.185001 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:35.167775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57704 (local address 127.0.97.2:38299)
0620 03:05:35.168108 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:35.168117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:35.168190 (+ 73us) server_negotiation.cc:407] Connection header received
0620 03:05:35.168325 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:35.168333 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:35.168420 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:35.168550 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:35.170078 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:35.170879 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:35.172704 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:35.173173 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:35.173265 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:35.178495 (+ 5230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:35.178507 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:35.178522 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:35.178566 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:35.181791 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:35.182718 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:35.182725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:35.182728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:35.182891 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:35.183401 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:35.183405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:35.183410 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:35.183803 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:35.184101 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:35.184467 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 03:05:35.184638 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260620 03:05:35.185729 1853 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57704
W20260620 03:05:35.186424 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 1723: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:05:35.186589 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 1723
I20260620 03:05:35.202864 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 1862
Jun 20 03:05:35 dist-test-slave-xv68 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:05:35.217809 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:35.203564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.0.97.3:37009)
0620 03:05:35.204072 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:35.204077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:35.204093 (+ 16us) server_negotiation.cc:407] Connection header received
0620 03:05:35.204147 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:35.204151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:35.204212 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:35.204313 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:35.205601 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:35.206444 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:35.207776 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:35.208192 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:35.208300 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:35.212358 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:35.212372 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:35.212388 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:35.212420 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:35.215200 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:35.216009 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:35.216017 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:35.216019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:35.216153 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:35.216510 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:35.216513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:35.216515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:35.216816 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:35.217039 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:35.217379 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 03:05:35.217544 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":306,"threads_started":1}
W20260620 03:05:35.218503 1992 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37488
W20260620 03:05:35.219245 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 1862: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:05:35.219480 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 1862
I20260620 03:05:35.237195 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 1480
I20260620 03:05:35.238251 1552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:05:35.351431 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 1480
2026-06-20T03:05:35Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4950 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:05:35 dist-test-slave-xv68 krb5kdc[2102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:05:35 dist-test-slave-xv68 krb5kdc[2102](info): set up 2 sockets
Jun 20 03:05:35 dist-test-slave-xv68 krb5kdc[2102](info): commencing operation
krb5kdc: starting...
W20260620 03:05:36.439433 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:05:36 dist-test-slave-xv68 krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:05:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:05:36.623154 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45311
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:36.940071 2115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:36.940346 2115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:36.940532 2115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:36.947778 2115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:05:36.947909 2115 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 03:05:36.947983 2115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:36.948091 2115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:36.948176 2115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:05:36.948284 2115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:05:36.957996 2115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45311
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:36.959276 2115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:36.960625 2115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:36.972198 2120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:36.972281 2121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:36.972242 2123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:36.973471 2115 server_base.cc:1034] running on GCE node
I20260620 03:05:36.974202 2115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:36.975633 2115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:36.976850 2115 hybrid_clock.cc:648] HybridClock initialized: now 1781924736976789 us; error 32 us; skew 500 ppm
I20260620 03:05:36.977205 2115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:36 dist-test-slave-xv68 krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:36.980240 2115 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:05:36.981722 2115 webserver.cc:460] Webserver started at http://127.0.97.62:43763/ using document root <none> and password file <none>
I20260620 03:05:36.982429 2115 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:36.982559 2115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:36.982882 2115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:36.985893 2115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "07ce8814fbb84557a1053a589b6bd10e"
format_stamp: "Formatted at 2026-06-20 03:05:36 on dist-test-slave-xv68"
server_key: "36d12f0b1ec7e2771fa32f089459dc6f"
server_key_iv: "f3172494d435f9ea9f246fbd6a5ebf5f"
server_key_version: "encryptionkey@0"
I20260620 03:05:36.986666 2115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "07ce8814fbb84557a1053a589b6bd10e"
format_stamp: "Formatted at 2026-06-20 03:05:36 on dist-test-slave-xv68"
server_key: "36d12f0b1ec7e2771fa32f089459dc6f"
server_key_iv: "f3172494d435f9ea9f246fbd6a5ebf5f"
server_key_version: "encryptionkey@0"
I20260620 03:05:36.991683 2115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 03:05:36.995148 2130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:36.996483 2115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:05:36.996681 2115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "07ce8814fbb84557a1053a589b6bd10e"
format_stamp: "Formatted at 2026-06-20 03:05:36 on dist-test-slave-xv68"
server_key: "36d12f0b1ec7e2771fa32f089459dc6f"
server_key_iv: "f3172494d435f9ea9f246fbd6a5ebf5f"
server_key_version: "encryptionkey@0"
I20260620 03:05:36.996914 2115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:37.023727 2115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:37.027909 2115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:37.028198 2115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:37.050410 2115 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45311
I20260620 03:05:37.050462 2191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45311 every 8 connection(s)
I20260620 03:05:37.051914 2115 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:05:37.055528 2192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:37.059465 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 2115
I20260620 03:05:37.059656 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:05:37.060420 388 external_mini_cluster.cc:1428] Setting key 1cfb052134edc85d35890522be73f645
I20260620 03:05:37.066732 2192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e: Bootstrap starting.
I20260620 03:05:37.070169 2192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:37.071376 2192 log.cc:824] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:37.074630 2192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e: No bootstrap required, opened a new log
Jun 20 03:05:37 dist-test-slave-xv68 krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:37.085109 2192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ce8814fbb84557a1053a589b6bd10e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45311 } }
I20260620 03:05:37.085562 2192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:37.085672 2192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ce8814fbb84557a1053a589b6bd10e, State: Initialized, Role: FOLLOWER
I20260620 03:05:37.086107 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:37.062514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.0.97.62:45311)
0620 03:05:37.063421 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:37.063432 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:37.063480 (+ 48us) server_negotiation.cc:407] Connection header received
0620 03:05:37.065553 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:37.065575 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:37.065951 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:37.066289 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:37.067997 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:37.069407 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:37.071150 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:37.072182 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:37.072405 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:37.077153 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:37.077164 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:37.077183 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:37.077244 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:37.081487 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:37.082200 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:37.082209 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:37.082218 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:37.082454 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:37.082792 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:37.082797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:37.082801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:37.083583 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:37.083882 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:37.084242 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 03:05:37.084688 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":419,"threads_started":1}
I20260620 03:05:37.086369 2192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ce8814fbb84557a1053a589b6bd10e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45311 } }
I20260620 03:05:37.086664 2192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:05:37.086822 2192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:05:37.087013 2192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:37.089289 2192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ce8814fbb84557a1053a589b6bd10e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45311 } }
I20260620 03:05:37.089769 2192 leader_election.cc:302] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07ce8814fbb84557a1053a589b6bd10e; no voters:
I20260620 03:05:37.090337 2192 leader_election.cc:288] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:05:37.090487 2197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:05:37.092216 2192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:05:37.097383 2197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e [term 1 LEADER]: Becoming Leader. State: Replica: 07ce8814fbb84557a1053a589b6bd10e, State: Running, Role: LEADER
I20260620 03:05:37.097856 2197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ce8814fbb84557a1053a589b6bd10e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45311 } }
I20260620 03:05:37.100620 2198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07ce8814fbb84557a1053a589b6bd10e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ce8814fbb84557a1053a589b6bd10e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45311 } } }
I20260620 03:05:37.100817 2199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07ce8814fbb84557a1053a589b6bd10e. Latest consensus state: current_term: 1 leader_uuid: "07ce8814fbb84557a1053a589b6bd10e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ce8814fbb84557a1053a589b6bd10e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45311 } } }
I20260620 03:05:37.101001 2198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e [sys.catalog]: This master's current role is: LEADER
I20260620 03:05:37.101624 2199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e [sys.catalog]: This master's current role is: LEADER
I20260620 03:05:37.101828 2212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:05:37.107038 2212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:05:37.115622 2212 catalog_manager.cc:1347] Generated new cluster ID: 207738c33cf8434d86bd5bbfffaa61fe
I20260620 03:05:37.115727 2212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:05:37.130012 2212 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:05:37.131549 2212 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:05:37.156723 2212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e: Generated new TSK 0
I20260620 03:05:37.157636 2212 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:05:37.218915 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45311
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:37.547535 2218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:37.547794 2218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:37.547896 2218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:37.554854 2218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:37.555007 2218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:37.555090 2218 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 03:05:37.555142 2218 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 03:05:37.555203 2218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:05:37.564443 2218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:37.565598 2218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:37.566843 2218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:37.579156 2223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:37.579231 2224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:37.579200 2226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:37.580042 2218 server_base.cc:1034] running on GCE node
I20260620 03:05:37.580729 2218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:37.581729 2218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:37.582932 2218 hybrid_clock.cc:648] HybridClock initialized: now 1781924737582865 us; error 59 us; skew 500 ppm
I20260620 03:05:37.583257 2218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:37 dist-test-slave-xv68 krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:37.586481 2218 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:05:37.588122 2218 webserver.cc:460] Webserver started at http://127.0.97.1:36323/ using document root <none> and password file <none>
I20260620 03:05:37.588757 2218 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:37.588855 2218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:37.589104 2218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:37.592068 2218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "c60ce80299644581886852102ca91fa0"
format_stamp: "Formatted at 2026-06-20 03:05:37 on dist-test-slave-xv68"
server_key: "93487cca01d8007f3d5c2c7d652c9adf"
server_key_iv: "dbf0b254c1fb961c3815661bb2e605a3"
server_key_version: "encryptionkey@0"
I20260620 03:05:37.592865 2218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c60ce80299644581886852102ca91fa0"
format_stamp: "Formatted at 2026-06-20 03:05:37 on dist-test-slave-xv68"
server_key: "93487cca01d8007f3d5c2c7d652c9adf"
server_key_iv: "dbf0b254c1fb961c3815661bb2e605a3"
server_key_version: "encryptionkey@0"
I20260620 03:05:37.598269 2218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:05:37.602084 2233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:37.603652 2218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:05:37.603829 2218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "c60ce80299644581886852102ca91fa0"
format_stamp: "Formatted at 2026-06-20 03:05:37 on dist-test-slave-xv68"
server_key: "93487cca01d8007f3d5c2c7d652c9adf"
server_key_iv: "dbf0b254c1fb961c3815661bb2e605a3"
server_key_version: "encryptionkey@0"
I20260620 03:05:37.604039 2218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:37.617012 2218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:37.621006 2218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:37.621308 2218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:37.623951 2218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:37.624068 2218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:37.624199 2218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:37.624328 2218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 20 03:05:37 dist-test-slave-xv68 krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:37.647540 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:37.624424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53165 (local address 127.0.97.62:45311)
0620 03:05:37.625276 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:37.625281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:37.626148 (+ 867us) server_negotiation.cc:407] Connection header received
0620 03:05:37.628200 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:37.628205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:37.628290 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:37.628421 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:37.630672 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:37.631711 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:37.634958 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:37.635457 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:37.635594 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:37.640778 (+ 5184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:37.640783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:37.640786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:37.640817 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:37.643591 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:37.644706 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:37.644712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:37.644718 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:37.644869 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:37.645441 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:37.645449 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:37.645454 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:37.645864 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:37.646026 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:37.646685 (+ 659us) server_negotiation.cc:299] Negotiation successful
0620 03:05:37.646971 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":651,"threads_started":1}
I20260620 03:05:37.648156 2243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:37.625106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45311 (local address 127.0.97.1:53165)
0620 03:05:37.625965 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:37.626011 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:37.627912 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:37.628703 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:37.628711 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:37.629492 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:37.630462 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:37.630494 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:37.631909 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:37.631915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:37.634813 (+ 2898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:37.634828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:37.635786 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:37.635796 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:37.636156 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:37.637098 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:37.637137 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:37.640525 (+ 3388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:37.643754 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:37.643768 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:37.643786 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:37.644550 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:37.644986 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:37.644991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:37.644995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:37.645288 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:37.646017 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:37.646023 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:37.646324 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 03:05:37.646550 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 03:05:37.646923 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":422,"threads_started":1}
I20260620 03:05:37.671322 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:37.657522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60931 (local address 127.0.97.62:45311)
0620 03:05:37.657701 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:37.657709 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:37.658557 (+ 848us) server_negotiation.cc:407] Connection header received
0620 03:05:37.658764 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:37.658771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:37.658835 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:37.658936 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:37.660260 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:37.661293 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:37.662836 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:37.663222 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:37.663381 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:37.665835 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:37.665849 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:37.665853 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:37.665902 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:37.668847 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:37.669478 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:37.669485 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:37.669490 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:37.669612 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:37.669956 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:37.669962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:37.669966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:37.670200 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:37.670343 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:37.670738 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 03:05:37.670948 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 03:05:37.671424 2305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:37.657617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45311 (local address 127.0.97.1:60931)
0620 03:05:37.658452 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:37.658467 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:37.658628 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:37.659147 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:37.659150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:37.659690 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:37.660121 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:37.660131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:37.661434 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:37.661445 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:37.662704 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:37.662716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:37.663435 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:37.663442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:37.663612 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:37.664423 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:37.664440 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:37.665589 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:37.668971 (+ 3382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:37.668976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:37.668980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:37.669367 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:37.669724 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:37.669729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:37.669733 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:37.669856 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:37.670308 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:37.670313 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:37.670489 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 03:05:37.670638 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 03:05:37.670926 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":588,"threads_started":1}
I20260620 03:05:37.679594 2218 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44911
I20260620 03:05:37.679618 2353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44911 every 8 connection(s)
I20260620 03:05:37.680764 2218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:05:37.682554 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 2218
I20260620 03:05:37.682757 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:05:37.682894 2354 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45311
I20260620 03:05:37.683115 2354 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:37.683523 388 external_mini_cluster.cc:1428] Setting key b96256e02bf22a55177606574f06b0f5
I20260620 03:05:37.683857 2354 heartbeater.cc:511] Master 127.0.97.62:45311 requested a full tablet report, sending...
I20260620 03:05:37.685791 2146 ts_manager.cc:194] Registered new tserver with Master: c60ce80299644581886852102ca91fa0 (127.0.97.1:44911)
I20260620 03:05:37.687209 2146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53165
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:05:37.722955 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45311
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:38.042039 2358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:38.042306 2358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:38.042407 2358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:38.049330 2358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:38.049422 2358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:38.049566 2358 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 03:05:38.049625 2358 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 03:05:38.049759 2358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:05:38.059259 2358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:38.060494 2358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:38.061726 2358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:38.074259 2363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:38.074321 2364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:38.074354 2366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:38.075479 2358 server_base.cc:1034] running on GCE node
I20260620 03:05:38.076033 2358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:38.076843 2358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:38.078039 2358 hybrid_clock.cc:648] HybridClock initialized: now 1781924738078008 us; error 30 us; skew 500 ppm
I20260620 03:05:38.078336 2358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:38 dist-test-slave-xv68 krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:38.081401 2358 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:05:38.082886 2358 webserver.cc:460] Webserver started at http://127.0.97.2:39409/ using document root <none> and password file <none>
I20260620 03:05:38.083613 2358 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:38.083755 2358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:38.084060 2358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:38.086833 2358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "2882899786974c498b4f4c43442563bd"
format_stamp: "Formatted at 2026-06-20 03:05:38 on dist-test-slave-xv68"
server_key: "cb74781535ffcb1799bd5fa763e1c12b"
server_key_iv: "161e701c83536785b61ce472e72c5b4d"
server_key_version: "encryptionkey@0"
I20260620 03:05:38.087697 2358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2882899786974c498b4f4c43442563bd"
format_stamp: "Formatted at 2026-06-20 03:05:38 on dist-test-slave-xv68"
server_key: "cb74781535ffcb1799bd5fa763e1c12b"
server_key_iv: "161e701c83536785b61ce472e72c5b4d"
server_key_version: "encryptionkey@0"
I20260620 03:05:38.092808 2358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 03:05:38.096297 2373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:38.097628 2358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:05:38.097796 2358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "2882899786974c498b4f4c43442563bd"
format_stamp: "Formatted at 2026-06-20 03:05:38 on dist-test-slave-xv68"
server_key: "cb74781535ffcb1799bd5fa763e1c12b"
server_key_iv: "161e701c83536785b61ce472e72c5b4d"
server_key_version: "encryptionkey@0"
I20260620 03:05:38.097992 2358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:38.122608 2358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:38.126778 2358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:38.127069 2358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:38.129711 2358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:38.129838 2358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:38.129994 2358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:38.130091 2358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 03:05:38 dist-test-slave-xv68 krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:38.154959 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.130603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42687 (local address 127.0.97.62:45311)
0620 03:05:38.130823 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:38.130830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:38.132432 (+ 1602us) server_negotiation.cc:407] Connection header received
0620 03:05:38.135651 (+ 3219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:38.135656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:38.135741 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:38.135809 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:38.138715 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.139872 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.143039 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.143599 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.143748 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.148063 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:38.148068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:38.148071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:38.148101 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:38.151472 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:38.152418 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:38.152427 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:38.152433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:38.152588 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:38.153250 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:38.153255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:38.153261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:38.153551 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:38.153699 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:38.154402 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 03:05:38.154612 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 03:05:38.155994 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.131219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45311 (local address 127.0.97.2:42687)
0620 03:05:38.132242 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:38.132287 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:38.135263 (+ 2976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:38.136366 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:38.136382 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:38.137385 (+ 1003us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:38.138467 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.138497 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.140056 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.140064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.142870 (+ 2806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.142890 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.143874 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.143882 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.144086 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.144822 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:38.144843 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:38.147835 (+ 2992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:38.151604 (+ 3769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:38.151616 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:38.151636 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:38.152242 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:38.152712 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:38.152720 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:38.152725 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:38.153111 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:38.153690 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:38.153698 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:38.154038 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 03:05:38.154314 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 03:05:38.154805 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":392,"threads_started":1}
I20260620 03:05:38.179417 2358 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38757
I20260620 03:05:38.179452 2492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38757 every 8 connection(s)
I20260620 03:05:38.180962 2453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.164544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45311 (local address 127.0.97.2:54125)
0620 03:05:38.165324 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:38.165347 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:38.165587 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:38.166171 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:38.166182 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:38.167019 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:38.167650 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.167672 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.169348 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.169355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.170854 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.170871 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.172036 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.172046 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.172297 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.173144 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:38.173166 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:38.173861 (+ 695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:38.177995 (+ 4134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:38.178003 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:38.178010 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:38.178670 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:38.179136 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:38.179142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:38.179160 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:38.179372 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:38.180082 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:38.180086 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:38.180280 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 03:05:38.180393 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 03:05:38.180630 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":407,"threads_started":1}
I20260620 03:05:38.181219 2358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:05:38.181044 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.164461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54125 (local address 127.0.97.62:45311)
0620 03:05:38.164637 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:38.164646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:38.165442 (+ 796us) server_negotiation.cc:407] Connection header received
0620 03:05:38.165762 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:38.165769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:38.165865 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:38.165984 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:38.167875 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.169178 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.171041 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.171818 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.171979 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.174063 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:38.174071 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:38.174079 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:38.174126 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:38.177804 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:38.178813 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:38.178823 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:38.178829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:38.178989 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:38.179493 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:38.179501 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:38.179507 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:38.179939 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:38.180084 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:38.180518 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 03:05:38.180750 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 03:05:38.183386 2493 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45311
I20260620 03:05:38.184556 2493 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:38.185314 2493 heartbeater.cc:511] Master 127.0.97.62:45311 requested a full tablet report, sending...
I20260620 03:05:38.186574 2146 ts_manager.cc:194] Registered new tserver with Master: 2882899786974c498b4f4c43442563bd (127.0.97.2:38757)
I20260620 03:05:38.187394 2146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42687
I20260620 03:05:38.190518 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 2358
I20260620 03:05:38.190692 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:05:38.191139 388 external_mini_cluster.cc:1428] Setting key e15e523f1fd5e13db397758d49cbeb01
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:05:38.231122 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45311
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:38.543231 2497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:38.543571 2497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:38.543752 2497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:38.550766 2497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:38.550908 2497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:38.551050 2497 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 03:05:38.551136 2497 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 03:05:38.551266 2497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:05:38.560684 2497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:38.561993 2497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:38.563275 2497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:38.577008 2503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:38.577118 2505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:38.577080 2502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:38.578094 2497 server_base.cc:1034] running on GCE node
I20260620 03:05:38.578862 2497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:38.579960 2497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:38.581176 2497 hybrid_clock.cc:648] HybridClock initialized: now 1781924738581108 us; error 60 us; skew 500 ppm
I20260620 03:05:38.581522 2497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:38 dist-test-slave-xv68 krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:38.584808 2497 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:05:38.586412 2497 webserver.cc:460] Webserver started at http://127.0.97.3:37649/ using document root <none> and password file <none>
I20260620 03:05:38.587111 2497 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:38.587256 2497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:38.587610 2497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:38.590412 2497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "b0ec74dff413420bbd1e9164226ae5c6"
format_stamp: "Formatted at 2026-06-20 03:05:38 on dist-test-slave-xv68"
server_key: "82164092b32a96697cd5712f15dc1686"
server_key_iv: "b4130837382e5344870aa1f7fb98ecba"
server_key_version: "encryptionkey@0"
I20260620 03:05:38.591162 2497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b0ec74dff413420bbd1e9164226ae5c6"
format_stamp: "Formatted at 2026-06-20 03:05:38 on dist-test-slave-xv68"
server_key: "82164092b32a96697cd5712f15dc1686"
server_key_iv: "b4130837382e5344870aa1f7fb98ecba"
server_key_version: "encryptionkey@0"
I20260620 03:05:38.596444 2497 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 03:05:38.599766 2512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:38.601154 2497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 03:05:38.601322 2497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "b0ec74dff413420bbd1e9164226ae5c6"
format_stamp: "Formatted at 2026-06-20 03:05:38 on dist-test-slave-xv68"
server_key: "82164092b32a96697cd5712f15dc1686"
server_key_iv: "b4130837382e5344870aa1f7fb98ecba"
server_key_version: "encryptionkey@0"
I20260620 03:05:38.601519 2497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:38.622151 2497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:38.625967 2497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:38.626252 2497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:38.628970 2497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:38.629091 2497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:38.629222 2497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:38.629297 2497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 03:05:38 dist-test-slave-xv68 krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:38.651031 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.629461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41293 (local address 127.0.97.62:45311)
0620 03:05:38.629695 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:38.629701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:38.631146 (+ 1445us) server_negotiation.cc:407] Connection header received
0620 03:05:38.633245 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:38.633251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:38.633339 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:38.633452 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:38.635606 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.636644 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.639238 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.639696 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.639830 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.644591 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:38.644596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:38.644600 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:38.644633 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:38.647381 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:38.648316 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:38.648325 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:38.648330 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:38.648457 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:38.648863 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:38.648868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:38.648872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:38.649131 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:38.649279 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:38.650179 (+ 900us) server_negotiation.cc:299] Negotiation successful
0620 03:05:38.650527 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 03:05:38.651862 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.630087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45311 (local address 127.0.97.3:41293)
0620 03:05:38.630960 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:38.631003 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:38.632958 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:38.633749 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:38.633765 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:38.634554 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:38.635434 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.635453 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.636824 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.636829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.639026 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.639047 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.639950 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.639959 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.640307 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.641126 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:38.641158 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:38.644373 (+ 3215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:38.647503 (+ 3130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:38.647513 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:38.647529 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:38.648180 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:38.648563 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:38.648569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:38.648573 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:38.648748 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:38.649258 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:38.649266 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:38.649623 (+ 357us) client_negotiation.cc:769] Sending connection context
0620 03:05:38.649973 (+ 350us) client_negotiation.cc:240] Negotiation successful
0620 03:05:38.650452 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":403,"threads_started":1}
I20260620 03:05:38.673486 2580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.658876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45311 (local address 127.0.97.3:51593)
0620 03:05:38.659575 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:38.659596 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:38.659717 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:38.660107 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:38.660110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:38.660520 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:38.660785 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.660793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.662597 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.662602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.663920 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.663941 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.665241 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.665247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.665385 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.666214 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:38.666231 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:38.667399 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:38.670845 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:38.670853 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:38.670857 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:38.671262 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:38.671683 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:38.671691 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:38.671695 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:38.671874 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:38.672661 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:38.672664 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:38.672829 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 03:05:38.672928 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 03:05:38.673163 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":427,"threads_started":1}
I20260620 03:05:38.674104 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.658908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51593 (local address 127.0.97.62:45311)
0620 03:05:38.659144 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:38.659152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:38.659690 (+ 538us) server_negotiation.cc:407] Connection header received
0620 03:05:38.659861 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:38.659866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:38.659964 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:38.660096 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:38.660923 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.662411 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.664149 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.664802 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.664961 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.667604 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:38.667609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:38.667612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:38.667641 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:38.670703 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:38.671427 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:38.671432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:38.671436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:38.671559 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:38.672033 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:38.672042 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:38.672048 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:38.672517 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:38.673107 (+ 590us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:38.673307 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 03:05:38.673605 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 03:05:38.689007 2497 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46439
I20260620 03:05:38.689051 2631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46439 every 8 connection(s)
I20260620 03:05:38.689790 2354 heartbeater.cc:503] Master 127.0.97.62:45311 was elected leader, sending a full tablet report...
I20260620 03:05:38.690347 2497 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:05:38.692389 2632 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45311
I20260620 03:05:38.692607 2632 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:38.693331 2632 heartbeater.cc:511] Master 127.0.97.62:45311 requested a full tablet report, sending...
I20260620 03:05:38.694654 2146 ts_manager.cc:194] Registered new tserver with Master: b0ec74dff413420bbd1e9164226ae5c6 (127.0.97.3:46439)
I20260620 03:05:38.695508 2146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41293
I20260620 03:05:38.698189 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 2497
I20260620 03:05:38.698331 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:05:38.698704 388 external_mini_cluster.cc:1428] Setting key a83c6ab89900bc4356ff5b053ff63cac
I20260620 03:05:38.701562 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 03:05:38 dist-test-slave-xv68 krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 03:05:38 dist-test-slave-xv68 krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:38.746773 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.734020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49166 (local address 127.0.97.62:45311)
0620 03:05:38.734243 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:38.734250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:38.734452 (+ 202us) server_negotiation.cc:407] Connection header received
0620 03:05:38.734599 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:38.734603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:38.734674 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:38.734761 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:38.736289 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.737036 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.738201 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.738593 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.738737 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.742275 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:38.742280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:38.742283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:38.742314 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:38.744720 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:38.745275 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:38.745279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:38.745281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:38.745383 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:38.745679 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:38.745682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:38.745684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:38.745931 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:38.746101 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:38.746345 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 03:05:38.746501 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 03:05:38.750519 2146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:05:38.758401 2146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:05:38.780413 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.769369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60844 (local address 127.0.97.2:38757)
0620 03:05:38.771929 (+ 2560us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:38.771940 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:38.771969 (+ 29us) server_negotiation.cc:407] Connection header received
0620 03:05:38.772070 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:38.772079 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:38.772341 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:38.772849 (+ 508us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:38.773610 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.775110 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.777775 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.779207 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.779368 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.779586 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:38.779942 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 03:05:38.780112 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2409,"thread_start_us":2311,"threads_started":1}
I20260620 03:05:38.781257 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.769220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38757 (local address 127.0.0.1:60844)
0620 03:05:38.771044 (+ 1824us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:38.771089 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:38.771359 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:38.772882 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:38.772889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:38.772917 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:38.773282 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.773295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.775320 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.775329 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.777590 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.777606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.779451 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.779463 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.779675 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.779741 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 03:05:38.779884 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 03:05:38.780074 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1453,"thread_start_us":1541,"threads_started":1}
I20260620 03:05:38.781627 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.770281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46439 (local address 127.0.0.1:34328)
0620 03:05:38.772200 (+ 1919us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:38.772216 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:38.772315 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:38.774716 (+ 2401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:38.774724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:38.774760 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:38.775363 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.775385 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.777451 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.777457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.778874 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.778886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.780891 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.780897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.781076 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.781110 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 03:05:38.781173 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 03:05:38.781328 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1823,"thread_start_us":4779,"threads_started":1}
I20260620 03:05:38.782220 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.770850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34328 (local address 127.0.97.3:46439)
0620 03:05:38.774139 (+ 3289us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:38.774147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:38.774177 (+ 30us) server_negotiation.cc:407] Connection header received
0620 03:05:38.774260 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:38.774267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:38.774498 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:38.774687 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:38.775625 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.777296 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.779037 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.780410 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.781500 (+ 1090us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.781765 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:38.781839 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 03:05:38.781997 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3127,"thread_start_us":2982,"threads_started":1}
I20260620 03:05:38.784348 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.771675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44911 (local address 127.0.0.1:41318)
0620 03:05:38.775389 (+ 3714us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:38.775406 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:38.775609 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:38.776410 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:38.776414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:38.776440 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:38.776874 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.776885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.778643 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.778648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.780659 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.780676 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.783583 (+ 2907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.783593 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.783767 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.783803 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 03:05:38.783907 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 03:05:38.784047 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3554,"thread_start_us":698,"threads_started":1}
I20260620 03:05:38.784395 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.772189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41318 (local address 127.0.97.1:44911)
0620 03:05:38.772755 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:38.772766 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:38.775499 (+ 2733us) server_negotiation.cc:407] Connection header received
0620 03:05:38.775781 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:38.775792 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:38.776030 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:38.776216 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:38.777077 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.778436 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.780815 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.783393 (+ 2578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.783520 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.783698 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:38.783956 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 03:05:38.784110 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":290,"threads_started":1}
I20260620 03:05:38.785312 2423 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd60c1d71cd4c6093d2084e73a21943 (DEFAULT_TABLE table=test-table [id=9fc3f9257ed644618a641eb81ba8925f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:38.785610 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd60c1d71cd4c6093d2084e73a21943 (DEFAULT_TABLE table=test-table [id=9fc3f9257ed644618a641eb81ba8925f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:38.786432 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd60c1d71cd4c6093d2084e73a21943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:38.786953 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd60c1d71cd4c6093d2084e73a21943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:38.787644 2284 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd60c1d71cd4c6093d2084e73a21943 (DEFAULT_TABLE table=test-table [id=9fc3f9257ed644618a641eb81ba8925f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:38.788861 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd60c1d71cd4c6093d2084e73a21943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:38.798069 2651 tablet_bootstrap.cc:492] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd: Bootstrap starting.
I20260620 03:05:38.801080 2651 tablet_bootstrap.cc:654] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:38.802212 2652 tablet_bootstrap.cc:492] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6: Bootstrap starting.
I20260620 03:05:38.802457 2651 log.cc:824] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:38.806706 2652 tablet_bootstrap.cc:654] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:38.808393 2652 log.cc:824] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:38.809648 2653 tablet_bootstrap.cc:492] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0: Bootstrap starting.
I20260620 03:05:38.812031 2652 tablet_bootstrap.cc:492] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6: No bootstrap required, opened a new log
I20260620 03:05:38.812547 2652 ts_tablet_manager.cc:1397] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 03:05:38.812597 2653 tablet_bootstrap.cc:654] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:38.813692 2653 log.cc:824] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:38.815083 2651 tablet_bootstrap.cc:492] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd: No bootstrap required, opened a new log
I20260620 03:05:38.815378 2651 ts_tablet_manager.cc:1397] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20260620 03:05:38.816558 2653 tablet_bootstrap.cc:492] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0: No bootstrap required, opened a new log
I20260620 03:05:38.816804 2653 ts_tablet_manager.cc:1397] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 03:05:38.820194 2652 raft_consensus.cc:348] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:38.820537 2652 raft_consensus.cc:374] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:38.820655 2652 raft_consensus.cc:729] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ec74dff413420bbd1e9164226ae5c6, State: Initialized, Role: FOLLOWER
I20260620 03:05:38.821254 2652 consensus_queue.cc:260] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:38.822271 2632 heartbeater.cc:503] Master 127.0.97.62:45311 was elected leader, sending a full tablet report...
I20260620 03:05:38.824642 2652 ts_tablet_manager.cc:1428] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20260620 03:05:38.826011 2651 raft_consensus.cc:348] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:38.826426 2651 raft_consensus.cc:374] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:38.826522 2651 raft_consensus.cc:729] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2882899786974c498b4f4c43442563bd, State: Initialized, Role: FOLLOWER
I20260620 03:05:38.827133 2651 consensus_queue.cc:260] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:38.827034 2653 raft_consensus.cc:348] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:38.827468 2653 raft_consensus.cc:374] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:38.827574 2653 raft_consensus.cc:729] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60ce80299644581886852102ca91fa0, State: Initialized, Role: FOLLOWER
I20260620 03:05:38.828258 2653 consensus_queue.cc:260] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:38.828500 2493 heartbeater.cc:503] Master 127.0.97.62:45311 was elected leader, sending a full tablet report...
I20260620 03:05:38.828969 2651 ts_tablet_manager.cc:1428] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260620 03:05:38.829836 2653 ts_tablet_manager.cc:1428] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 03:05:38.872714 2659 raft_consensus.cc:482] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:05:38.873041 2659 raft_consensus.cc:504] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:38.874696 2659 leader_election.cc:288] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2882899786974c498b4f4c43442563bd (127.0.97.2:38757), b0ec74dff413420bbd1e9164226ae5c6 (127.0.97.3:46439)
I20260620 03:05:38.881690 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.875165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49401 (local address 127.0.97.3:46439)
0620 03:05:38.875390 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:38.875398 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:38.876330 (+ 932us) server_negotiation.cc:407] Connection header received
0620 03:05:38.876512 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:38.876516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:38.876571 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:38.876679 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:38.877439 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.878167 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.879968 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.880723 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.880855 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.880979 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:38.881302 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 03:05:38.881431 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 03:05:38.881713 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.875112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35075 (local address 127.0.97.2:38757)
0620 03:05:38.875354 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:38.875364 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:38.876210 (+ 846us) server_negotiation.cc:407] Connection header received
0620 03:05:38.876469 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:38.876477 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:38.876549 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:38.876612 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:38.877273 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.878057 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.880006 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.880730 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.880898 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.881001 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:38.881334 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 03:05:38.881453 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:05:38.881837 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.875189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46439 (local address 127.0.97.1:49401)
0620 03:05:38.876227 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:38.876242 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:38.876357 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:38.876709 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:38.876714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:38.876738 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:38.877250 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.877261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.878299 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.878302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.879807 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.879821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.880948 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.880955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.881143 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.881167 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 03:05:38.881240 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 03:05:38.881390 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"mutex_wait_us":30,"thread_start_us":553,"threads_started":1}
I20260620 03:05:38.881971 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.875064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38757 (local address 127.0.97.1:35075)
0620 03:05:38.876089 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:38.876106 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:38.876298 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:38.876656 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:38.876699 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:38.876723 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:38.877126 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.877136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.878182 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.878187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.879825 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.879837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.880990 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.880998 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.881157 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.881183 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 03:05:38.881291 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 03:05:38.881421 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":54,"thread_start_us":536,"threads_started":1}
I20260620 03:05:38.882988 2447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd60c1d71cd4c6093d2084e73a21943" candidate_uuid: "c60ce80299644581886852102ca91fa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2882899786974c498b4f4c43442563bd" is_pre_election: true
I20260620 03:05:38.882984 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd60c1d71cd4c6093d2084e73a21943" candidate_uuid: "c60ce80299644581886852102ca91fa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ec74dff413420bbd1e9164226ae5c6" is_pre_election: true
I20260620 03:05:38.883493 2447 raft_consensus.cc:2454] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c60ce80299644581886852102ca91fa0 in term 0.
I20260620 03:05:38.883493 2587 raft_consensus.cc:2454] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c60ce80299644581886852102ca91fa0 in term 0.
I20260620 03:05:38.884141 2235 leader_election.cc:302] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2882899786974c498b4f4c43442563bd, b0ec74dff413420bbd1e9164226ae5c6, c60ce80299644581886852102ca91fa0; no voters:
I20260620 03:05:38.884542 2659 raft_consensus.cc:2781] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:05:38.884707 2659 raft_consensus.cc:482] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:05:38.884838 2659 raft_consensus.cc:3037] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:38.887007 2659 raft_consensus.cc:504] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:38.887630 2659 leader_election.cc:288] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [CANDIDATE]: Term 1 election: Requested vote from peers 2882899786974c498b4f4c43442563bd (127.0.97.2:38757), b0ec74dff413420bbd1e9164226ae5c6 (127.0.97.3:46439)
I20260620 03:05:38.888052 2447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd60c1d71cd4c6093d2084e73a21943" candidate_uuid: "c60ce80299644581886852102ca91fa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2882899786974c498b4f4c43442563bd"
I20260620 03:05:38.888074 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd60c1d71cd4c6093d2084e73a21943" candidate_uuid: "c60ce80299644581886852102ca91fa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ec74dff413420bbd1e9164226ae5c6"
I20260620 03:05:38.888250 2447 raft_consensus.cc:3037] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:38.888263 2587 raft_consensus.cc:3037] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:38.890334 2447 raft_consensus.cc:2454] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c60ce80299644581886852102ca91fa0 in term 1.
I20260620 03:05:38.890370 2587 raft_consensus.cc:2454] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c60ce80299644581886852102ca91fa0 in term 1.
I20260620 03:05:38.890740 2236 leader_election.cc:302] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2882899786974c498b4f4c43442563bd, c60ce80299644581886852102ca91fa0; no voters:
I20260620 03:05:38.891040 2659 raft_consensus.cc:2781] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:05:38.891477 2659 raft_consensus.cc:686] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [term 1 LEADER]: Becoming Leader. State: Replica: c60ce80299644581886852102ca91fa0, State: Running, Role: LEADER
I20260620 03:05:38.891932 2659 consensus_queue.cc:237] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:38.896641 2147 catalog_manager.cc:5495] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c60ce80299644581886852102ca91fa0 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c60ce80299644581886852102ca91fa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } health_report { overall_health: UNKNOWN } } }
W20260620 03:05:38.932969 2355 tablet.cc:2369] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:05:38.934137 2494 tablet.cc:2369] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:05:38.942113 2633 tablet.cc:2369] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 03:05:38 dist-test-slave-xv68 krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:38.974392 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.961665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45311 (local address 127.0.0.1:49176)
0620 03:05:38.962229 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:38.962249 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:38.962424 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:38.962731 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:38.962735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:38.963519 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:38.963781 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.963790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.964811 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.964819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.965771 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:38.965781 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.966489 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.966495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:38.966651 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.967268 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:38.967284 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:38.969309 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:38.972219 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:38.972229 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:38.972235 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:38.972660 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:38.973004 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:38.973007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:38.973009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:38.973140 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:38.973597 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:38.973603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:38.973744 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 03:05:38.973878 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 03:05:38.974081 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":319,"threads_started":1}
I20260620 03:05:38.974480 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:38.961628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.0.97.62:45311)
0620 03:05:38.961819 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:38.961823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:38.962345 (+ 522us) server_negotiation.cc:407] Connection header received
0620 03:05:38.962544 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:38.962550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:38.962611 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:38.962709 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:38.963965 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.964683 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.965912 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:38.966308 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:38.966445 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:38.969480 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:38.969485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:38.969488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:38.969513 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:38.972103 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:38.972762 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:38.972766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:38.972768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:38.972911 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:38.973235 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:38.973238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:38.973240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:38.973492 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:38.973594 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:38.973966 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 03:05:38.974198 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":68}
I20260620 03:05:38.979203 2147 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49176:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 03:05:38.980511 2147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:05:38.987167 2284 tablet_service.cc:1467] Processing CreateTablet for tablet 8c99bcdf0d1e4175bb8cb2a21f0d23ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c0f9c64c133049ecab694877c175bb7a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 03:05:38.987242 2423 tablet_service.cc:1467] Processing CreateTablet for tablet 8c99bcdf0d1e4175bb8cb2a21f0d23ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c0f9c64c133049ecab694877c175bb7a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 03:05:38.987488 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 8c99bcdf0d1e4175bb8cb2a21f0d23ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c0f9c64c133049ecab694877c175bb7a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 03:05:38.988070 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c99bcdf0d1e4175bb8cb2a21f0d23ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:38.988143 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c99bcdf0d1e4175bb8cb2a21f0d23ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:38.988207 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c99bcdf0d1e4175bb8cb2a21f0d23ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:38.993491 2653 tablet_bootstrap.cc:492] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0: Bootstrap starting.
I20260620 03:05:38.995175 2652 tablet_bootstrap.cc:492] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6: Bootstrap starting.
I20260620 03:05:38.995175 2651 tablet_bootstrap.cc:492] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd: Bootstrap starting.
I20260620 03:05:38.995460 2653 tablet_bootstrap.cc:654] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:38.996752 2652 tablet_bootstrap.cc:654] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:38.996752 2651 tablet_bootstrap.cc:654] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:38.998127 2653 tablet_bootstrap.cc:492] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0: No bootstrap required, opened a new log
I20260620 03:05:38.998277 2653 ts_tablet_manager.cc:1397] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260620 03:05:38.999013 2653 raft_consensus.cc:348] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:38.999256 2653 raft_consensus.cc:374] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:38.999384 2653 raft_consensus.cc:729] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60ce80299644581886852102ca91fa0, State: Initialized, Role: FOLLOWER
I20260620 03:05:38.999634 2653 consensus_queue.cc:260] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:39.000056 2659 tablet_replica.cc:439] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } } }
I20260620 03:05:39.000324 2659 tablet_replica.cc:442] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:05:39.000907 2653 ts_tablet_manager.cc:1428] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 03:05:39.002642 2652 tablet_bootstrap.cc:492] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6: No bootstrap required, opened a new log
I20260620 03:05:39.002799 2652 ts_tablet_manager.cc:1397] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260620 03:05:39.003543 2651 tablet_bootstrap.cc:492] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd: No bootstrap required, opened a new log
I20260620 03:05:39.003733 2651 ts_tablet_manager.cc:1397] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 03:05:39.003597 2652 raft_consensus.cc:348] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:39.003865 2652 raft_consensus.cc:374] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:39.003974 2652 raft_consensus.cc:729] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ec74dff413420bbd1e9164226ae5c6, State: Initialized, Role: FOLLOWER
I20260620 03:05:39.004215 2652 consensus_queue.cc:260] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:39.004506 2651 raft_consensus.cc:348] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:39.004714 2651 raft_consensus.cc:374] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:39.004793 2652 ts_tablet_manager.cc:1428] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 03:05:39.004807 2651 raft_consensus.cc:729] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2882899786974c498b4f4c43442563bd, State: Initialized, Role: FOLLOWER
I20260620 03:05:39.004669 2657 tablet_replica.cc:439] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } } }
I20260620 03:05:39.005033 2657 tablet_replica.cc:442] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:05:39.005039 2651 consensus_queue.cc:260] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:39.005666 2651 ts_tablet_manager.cc:1428] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 03:05:39.005507 2658 tablet_replica.cc:439] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } } }
I20260620 03:05:39.005939 2658 tablet_replica.cc:442] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:05:39.193972 2657 raft_consensus.cc:482] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:05:39.194185 2657 raft_consensus.cc:504] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:39.195713 2657 leader_election.cc:288] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c60ce80299644581886852102ca91fa0 (127.0.97.1:44911), 2882899786974c498b4f4c43442563bd (127.0.97.2:38757)
I20260620 03:05:39.202668 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:39.196082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48777 (local address 127.0.97.1:44911)
0620 03:05:39.196253 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:39.196258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:39.196648 (+ 390us) server_negotiation.cc:407] Connection header received
0620 03:05:39.196926 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:39.196934 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:39.197024 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:39.197135 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:39.197742 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.198634 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.200531 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.201777 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.201903 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:39.201978 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:39.202326 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 03:05:39.202419 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 03:05:39.202917 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:39.196014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44911 (local address 127.0.97.3:48777)
0620 03:05:39.196540 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:39.196560 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:39.196698 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:39.197235 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:39.197241 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:39.197275 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:39.197567 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:39.197583 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.198792 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.198799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:39.200363 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:39.200380 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.201937 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.201944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:39.202148 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:39.202211 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 03:05:39.202306 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 03:05:39.202453 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"spinlock_wait_cycles":23808,"thread_start_us":297,"threads_started":1}
I20260620 03:05:39.203601 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:39.196721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38757 (local address 127.0.97.3:34839)
0620 03:05:39.197213 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:39.197226 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:39.197313 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:39.197649 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:39.197652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:39.197696 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:39.197983 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:39.197994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.199421 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.199425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:39.201095 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:39.201109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.202813 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.202829 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:39.203030 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:39.203061 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 03:05:39.203147 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 03:05:39.203278 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":268,"threads_started":1}
I20260620 03:05:39.203603 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:39.196959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34839 (local address 127.0.97.2:38757)
0620 03:05:39.197145 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:39.197151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:39.197284 (+ 133us) server_negotiation.cc:407] Connection header received
0620 03:05:39.197427 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:39.197433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:39.197520 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:39.197620 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:39.198233 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.199188 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.201240 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.202464 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.202578 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:39.202666 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:39.203196 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 03:05:39.203367 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 03:05:39.203917 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c99bcdf0d1e4175bb8cb2a21f0d23ad" candidate_uuid: "b0ec74dff413420bbd1e9164226ae5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60ce80299644581886852102ca91fa0" is_pre_election: true
I20260620 03:05:39.204303 2309 raft_consensus.cc:2454] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0ec74dff413420bbd1e9164226ae5c6 in term 0.
I20260620 03:05:39.204932 2515 leader_election.cc:302] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0ec74dff413420bbd1e9164226ae5c6, c60ce80299644581886852102ca91fa0; no voters:
I20260620 03:05:39.205143 2447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c99bcdf0d1e4175bb8cb2a21f0d23ad" candidate_uuid: "b0ec74dff413420bbd1e9164226ae5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2882899786974c498b4f4c43442563bd" is_pre_election: true
I20260620 03:05:39.205397 2657 raft_consensus.cc:2781] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:05:39.205433 2447 raft_consensus.cc:2454] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0ec74dff413420bbd1e9164226ae5c6 in term 0.
I20260620 03:05:39.205518 2657 raft_consensus.cc:482] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:05:39.205600 2657 raft_consensus.cc:3037] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:39.207773 2657 raft_consensus.cc:504] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:39.208344 2657 leader_election.cc:288] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [CANDIDATE]: Term 1 election: Requested vote from peers c60ce80299644581886852102ca91fa0 (127.0.97.1:44911), 2882899786974c498b4f4c43442563bd (127.0.97.2:38757)
I20260620 03:05:39.208763 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c99bcdf0d1e4175bb8cb2a21f0d23ad" candidate_uuid: "b0ec74dff413420bbd1e9164226ae5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60ce80299644581886852102ca91fa0"
I20260620 03:05:39.208792 2447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c99bcdf0d1e4175bb8cb2a21f0d23ad" candidate_uuid: "b0ec74dff413420bbd1e9164226ae5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2882899786974c498b4f4c43442563bd"
I20260620 03:05:39.208963 2309 raft_consensus.cc:3037] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:39.208979 2447 raft_consensus.cc:3037] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:39.211942 2309 raft_consensus.cc:2454] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0ec74dff413420bbd1e9164226ae5c6 in term 1.
I20260620 03:05:39.211927 2447 raft_consensus.cc:2454] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0ec74dff413420bbd1e9164226ae5c6 in term 1.
I20260620 03:05:39.212348 2515 leader_election.cc:302] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0ec74dff413420bbd1e9164226ae5c6, c60ce80299644581886852102ca91fa0; no voters:
I20260620 03:05:39.212687 2657 raft_consensus.cc:2781] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:05:39.213104 2657 raft_consensus.cc:686] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [term 1 LEADER]: Becoming Leader. State: Replica: b0ec74dff413420bbd1e9164226ae5c6, State: Running, Role: LEADER
I20260620 03:05:39.213569 2657 consensus_queue.cc:237] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:39.216774 2675 tablet_replica.cc:439] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6: TxnStatusTablet state changed. Reason: New leader b0ec74dff413420bbd1e9164226ae5c6. Latest consensus state: current_term: 1 leader_uuid: "b0ec74dff413420bbd1e9164226ae5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } } }
I20260620 03:05:39.217155 2675 tablet_replica.cc:442] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6: This TxnStatusTablet replica's current role is: LEADER
I20260620 03:05:39.217844 2677 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 03:05:39.218034 2677 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 03:05:39.218190 2147 catalog_manager.cc:5495] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0ec74dff413420bbd1e9164226ae5c6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b0ec74dff413420bbd1e9164226ae5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } health_report { overall_health: HEALTHY } } }
I20260620 03:05:39.244438 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:39.238463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46439 (local address 127.0.0.1:34336)
0620 03:05:39.238719 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:39.238737 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:39.238850 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:39.239392 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:39.239396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:39.239405 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 03:05:39.239556 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:39.239562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.241002 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.241006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:39.241967 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:39.241977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.242899 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.242903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:39.243049 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:39.243098 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 03:05:39.244064 (+ 966us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 03:05:39.244077 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 03:05:39.244127 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 03:05:39.244228 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 03:05:39.244570 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:39.238622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34336 (local address 127.0.97.3:46439)
0620 03:05:39.238828 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:39.238834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:39.238856 (+ 22us) server_negotiation.cc:407] Connection header received
0620 03:05:39.239040 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:39.239047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:39.239170 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:39.239364 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:05:39.239675 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.240863 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.242101 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.242752 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.242882 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:39.243260 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:05:39.243967 (+ 707us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:05:39.244112 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:39.244168 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 03:05:39.244253 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 03:05:39.252084 2309 raft_consensus.cc:1261] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0 [term 1 FOLLOWER]: Refusing update from remote peer b0ec74dff413420bbd1e9164226ae5c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:05:39.252190 2447 raft_consensus.cc:1261] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd [term 1 FOLLOWER]: Refusing update from remote peer b0ec74dff413420bbd1e9164226ae5c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:05:39.253067 2657 consensus_queue.cc:1035] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:05:39.253441 2675 consensus_queue.cc:1035] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:05:39.264374 2658 tablet_replica.cc:439] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd: TxnStatusTablet state changed. Reason: New leader b0ec74dff413420bbd1e9164226ae5c6. Latest consensus state: current_term: 1 leader_uuid: "b0ec74dff413420bbd1e9164226ae5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } } }
I20260620 03:05:39.264693 2658 tablet_replica.cc:442] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:05:39.266063 2659 tablet_replica.cc:439] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0: TxnStatusTablet state changed. Reason: New leader b0ec74dff413420bbd1e9164226ae5c6. Latest consensus state: current_term: 1 leader_uuid: "b0ec74dff413420bbd1e9164226ae5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } } }
I20260620 03:05:39.266394 2659 tablet_replica.cc:442] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:05:39.270016 2681 mvcc.cc:205] Tried to move back new op lower bound from 7298763731965931520 to 7298763731822272512. Current Snapshot: MvccSnapshot[applied={T|T < 7298763731965931520}]
I20260620 03:05:39.272149 2658 tablet_replica.cc:439] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0ec74dff413420bbd1e9164226ae5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } } }
I20260620 03:05:39.272368 2658 tablet_replica.cc:442] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P 2882899786974c498b4f4c43442563bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:05:39.273494 2682 mvcc.cc:205] Tried to move back new op lower bound from 7298763731965931520 to 7298763731822272512. Current Snapshot: MvccSnapshot[applied={T|T < 7298763731965931520 or (T in {7298763731965931520})}]
I20260620 03:05:39.276006 2657 tablet_replica.cc:439] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0ec74dff413420bbd1e9164226ae5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } } }
I20260620 03:05:39.276299 2657 tablet_replica.cc:442] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6: This TxnStatusTablet replica's current role is: LEADER
I20260620 03:05:39.277585 2657 tablet_replica.cc:439] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0ec74dff413420bbd1e9164226ae5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } } }
I20260620 03:05:39.277786 2657 tablet_replica.cc:442] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P b0ec74dff413420bbd1e9164226ae5c6: This TxnStatusTablet replica's current role is: LEADER
I20260620 03:05:39.284443 2659 tablet_replica.cc:439] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0ec74dff413420bbd1e9164226ae5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } } }
I20260620 03:05:39.284778 2659 tablet_replica.cc:442] T 8c99bcdf0d1e4175bb8cb2a21f0d23ad P c60ce80299644581886852102ca91fa0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:05:39.291338 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:39.285454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.0.97.1:44911)
0620 03:05:39.285615 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:39.285622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:39.285646 (+ 24us) server_negotiation.cc:407] Connection header received
0620 03:05:39.285728 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:39.285732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:39.285846 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:39.286001 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:05:39.286471 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.287633 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.288993 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.289615 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.289742 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:39.290092 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:05:39.290733 (+ 641us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:05:39.290871 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:39.290927 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 03:05:39.291007 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 03:05:39.306859 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:39.301035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41059 (local address 127.0.97.3:46439)
0620 03:05:39.301259 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:39.301267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:39.301880 (+ 613us) server_negotiation.cc:407] Connection header received
0620 03:05:39.302088 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:39.302097 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:39.302171 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:39.302270 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:05:39.302806 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.303580 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.304932 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.305491 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.305661 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:39.306115 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:05:39.306224 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:05:39.306321 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:39.306485 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 03:05:39.306558 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 03:05:39.307004 2693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:39.301018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46439 (local address 127.0.97.1:41059)
0620 03:05:39.301776 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:39.301792 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:39.301938 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:39.302296 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:39.302300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:39.302311 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 03:05:39.302663 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:39.302672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.303701 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.303709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:39.304782 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:39.304793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.305704 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.305712 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:39.305847 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:39.305901 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 03:05:39.306337 (+ 436us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 03:05:39.306349 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 03:05:39.306461 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 03:05:39.306591 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":382,"threads_started":1}
I20260620 03:05:39.316219 2447 raft_consensus.cc:1261] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd [term 1 FOLLOWER]: Refusing update from remote peer c60ce80299644581886852102ca91fa0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:05:39.316396 2587 raft_consensus.cc:1261] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6 [term 1 FOLLOWER]: Refusing update from remote peer c60ce80299644581886852102ca91fa0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:05:39.316823 2662 consensus_queue.cc:1035] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:05:39.317354 2662 consensus_queue.cc:1035] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:05:39.350899 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:39.343431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42699 (local address 127.0.97.1:44911)
0620 03:05:39.343590 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:39.343597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:39.344342 (+ 745us) server_negotiation.cc:407] Connection header received
0620 03:05:39.344564 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:39.344571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:39.344651 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:39.344711 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:05:39.345504 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.346589 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.348838 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.349495 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.349589 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:39.350255 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:05:39.350346 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:05:39.350424 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:39.350570 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 03:05:39.350625 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 03:05:39.351416 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:39.343587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44911 (local address 127.0.97.3:42699)
0620 03:05:39.344251 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:39.344276 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:39.344408 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:39.344904 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:39.344909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:39.344921 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 03:05:39.345334 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:39.345346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.346764 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.346770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:39.348660 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:39.348678 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.349779 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.349785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:39.349952 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:39.350046 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 03:05:39.350457 (+ 411us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 03:05:39.350468 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 03:05:39.350556 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 03:05:39.350691 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":313,"threads_started":1}
I20260620 03:05:39.388902 2147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:05:39.389158 2147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:05:39.392516 2147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e: Sending DeleteTablet for 3 replicas of tablet 8fd60c1d71cd4c6093d2084e73a21943
I20260620 03:05:39.393685 2423 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd60c1d71cd4c6093d2084e73a21943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:05:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60844
I20260620 03:05:39.394784 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd60c1d71cd4c6093d2084e73a21943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:05:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34328
I20260620 03:05:39.396025 2284 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd60c1d71cd4c6093d2084e73a21943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:05:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41318
I20260620 03:05:39.397421 2147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:05:39.398113 2147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:05:39.407805 2704 tablet_replica.cc:330] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd: stopping tablet replica
I20260620 03:05:39.408136 2702 tablet_replica.cc:330] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6: stopping tablet replica
I20260620 03:05:39.408263 2704 raft_consensus.cc:2229] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:05:39.408624 2704 raft_consensus.cc:2258] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:05:39.408767 2702 raft_consensus.cc:2229] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:05:39.409176 2702 raft_consensus.cc:2258] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:05:39.411016 2702 ts_tablet_manager.cc:1905] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:05:39.413144 2423 tablet_service.cc:1467] Processing CreateTablet for tablet ae32906ff8af493897a1c5683a6f3657 (DEFAULT_TABLE table=test-table [id=fe22d826eaca481592dfd89fa7031b8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:39.413573 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae32906ff8af493897a1c5683a6f3657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:39.413988 2283 tablet_service.cc:1467] Processing CreateTablet for tablet ae32906ff8af493897a1c5683a6f3657 (DEFAULT_TABLE table=test-table [id=fe22d826eaca481592dfd89fa7031b8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:39.414443 2283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae32906ff8af493897a1c5683a6f3657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:39.419667 2562 tablet_service.cc:1467] Processing CreateTablet for tablet ae32906ff8af493897a1c5683a6f3657 (DEFAULT_TABLE table=test-table [id=fe22d826eaca481592dfd89fa7031b8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:39.420020 2651 tablet_bootstrap.cc:492] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd: Bootstrap starting.
I20260620 03:05:39.420064 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae32906ff8af493897a1c5683a6f3657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:39.420776 2704 ts_tablet_manager.cc:1905] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:05:39.421639 2651 tablet_bootstrap.cc:654] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:39.428148 2704 ts_tablet_manager.cc:1918] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 03:05:39.428314 2704 log.cc:1196] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/8fd60c1d71cd4c6093d2084e73a21943
I20260620 03:05:39.428853 2704 ts_tablet_manager.cc:1939] T 8fd60c1d71cd4c6093d2084e73a21943 P 2882899786974c498b4f4c43442563bd: Deleting consensus metadata
I20260620 03:05:39.430653 2133 catalog_manager.cc:4841] TS 2882899786974c498b4f4c43442563bd (127.0.97.2:38757): tablet 8fd60c1d71cd4c6093d2084e73a21943 (table test-table [id=9fc3f9257ed644618a641eb81ba8925f]) successfully deleted
I20260620 03:05:39.431823 2653 tablet_bootstrap.cc:492] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0: Bootstrap starting.
I20260620 03:05:39.433629 2653 tablet_bootstrap.cc:654] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:39.436131 2651 tablet_bootstrap.cc:492] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd: No bootstrap required, opened a new log
I20260620 03:05:39.436372 2651 ts_tablet_manager.cc:1397] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.004s
I20260620 03:05:39.437541 2651 raft_consensus.cc:348] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:39.437963 2651 raft_consensus.cc:374] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:39.438095 2651 raft_consensus.cc:729] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2882899786974c498b4f4c43442563bd, State: Initialized, Role: FOLLOWER
I20260620 03:05:39.438486 2651 consensus_queue.cc:260] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:39.441457 2651 ts_tablet_manager.cc:1428] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260620 03:05:39.445673 2702 ts_tablet_manager.cc:1918] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 03:05:39.445984 2702 log.cc:1196] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/8fd60c1d71cd4c6093d2084e73a21943
I20260620 03:05:39.446651 2702 ts_tablet_manager.cc:1939] T 8fd60c1d71cd4c6093d2084e73a21943 P b0ec74dff413420bbd1e9164226ae5c6: Deleting consensus metadata
I20260620 03:05:39.448609 2132 catalog_manager.cc:4841] TS b0ec74dff413420bbd1e9164226ae5c6 (127.0.97.3:46439): tablet 8fd60c1d71cd4c6093d2084e73a21943 (table test-table [id=9fc3f9257ed644618a641eb81ba8925f]) successfully deleted
I20260620 03:05:39.451895 2652 tablet_bootstrap.cc:492] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6: Bootstrap starting.
I20260620 03:05:39.453581 2652 tablet_bootstrap.cc:654] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:39.455144 2653 tablet_bootstrap.cc:492] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0: No bootstrap required, opened a new log
I20260620 03:05:39.455423 2653 ts_tablet_manager.cc:1397] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.001s
I20260620 03:05:39.456390 2653 raft_consensus.cc:348] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:39.456688 2653 raft_consensus.cc:374] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:39.456776 2653 raft_consensus.cc:729] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60ce80299644581886852102ca91fa0, State: Initialized, Role: FOLLOWER
I20260620 03:05:39.457105 2653 consensus_queue.cc:260] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:39.461935 2653 ts_tablet_manager.cc:1428] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260620 03:05:39.462491 2703 tablet_replica.cc:330] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0: stopping tablet replica
I20260620 03:05:39.462852 2703 raft_consensus.cc:2229] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:05:39.463824 2703 raft_consensus.cc:2258] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:05:39.466665 2703 ts_tablet_manager.cc:1905] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:05:39.472761 2652 tablet_bootstrap.cc:492] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6: No bootstrap required, opened a new log
I20260620 03:05:39.472931 2652 ts_tablet_manager.cc:1397] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.004s
I20260620 03:05:39.473691 2652 raft_consensus.cc:348] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:39.473927 2652 raft_consensus.cc:374] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:39.473903 2703 ts_tablet_manager.cc:1918] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 03:05:39.474005 2652 raft_consensus.cc:729] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ec74dff413420bbd1e9164226ae5c6, State: Initialized, Role: FOLLOWER
I20260620 03:05:39.474054 2703 log.cc:1196] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/8fd60c1d71cd4c6093d2084e73a21943
I20260620 03:05:39.474278 2652 consensus_queue.cc:260] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:39.474601 2703 ts_tablet_manager.cc:1939] T 8fd60c1d71cd4c6093d2084e73a21943 P c60ce80299644581886852102ca91fa0: Deleting consensus metadata
I20260620 03:05:39.476904 2652 ts_tablet_manager.cc:1428] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 03:05:39.476930 2133 catalog_manager.cc:4841] TS c60ce80299644581886852102ca91fa0 (127.0.97.1:44911): tablet 8fd60c1d71cd4c6093d2084e73a21943 (table test-table [id=9fc3f9257ed644618a641eb81ba8925f]) successfully deleted
I20260620 03:05:39.593099 2685 raft_consensus.cc:482] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:05:39.593369 2685 raft_consensus.cc:504] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:39.594295 2685 leader_election.cc:288] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c60ce80299644581886852102ca91fa0 (127.0.97.1:44911), 2882899786974c498b4f4c43442563bd (127.0.97.2:38757)
I20260620 03:05:39.594883 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae32906ff8af493897a1c5683a6f3657" candidate_uuid: "b0ec74dff413420bbd1e9164226ae5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60ce80299644581886852102ca91fa0" is_pre_election: true
I20260620 03:05:39.594911 2446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae32906ff8af493897a1c5683a6f3657" candidate_uuid: "b0ec74dff413420bbd1e9164226ae5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2882899786974c498b4f4c43442563bd" is_pre_election: true
I20260620 03:05:39.595172 2309 raft_consensus.cc:2454] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0ec74dff413420bbd1e9164226ae5c6 in term 0.
I20260620 03:05:39.595181 2446 raft_consensus.cc:2454] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0ec74dff413420bbd1e9164226ae5c6 in term 0.
I20260620 03:05:39.595747 2515 leader_election.cc:302] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2882899786974c498b4f4c43442563bd, b0ec74dff413420bbd1e9164226ae5c6; no voters:
I20260620 03:05:39.596180 2685 raft_consensus.cc:2781] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:05:39.596316 2685 raft_consensus.cc:482] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:05:39.596421 2685 raft_consensus.cc:3037] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:39.599730 2685 raft_consensus.cc:504] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:39.600481 2685 leader_election.cc:288] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [CANDIDATE]: Term 1 election: Requested vote from peers c60ce80299644581886852102ca91fa0 (127.0.97.1:44911), 2882899786974c498b4f4c43442563bd (127.0.97.2:38757)
I20260620 03:05:39.601023 2446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae32906ff8af493897a1c5683a6f3657" candidate_uuid: "b0ec74dff413420bbd1e9164226ae5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2882899786974c498b4f4c43442563bd"
I20260620 03:05:39.601250 2446 raft_consensus.cc:3037] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:39.601222 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae32906ff8af493897a1c5683a6f3657" candidate_uuid: "b0ec74dff413420bbd1e9164226ae5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60ce80299644581886852102ca91fa0"
I20260620 03:05:39.601442 2309 raft_consensus.cc:3037] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:39.604542 2446 raft_consensus.cc:2454] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0ec74dff413420bbd1e9164226ae5c6 in term 1.
I20260620 03:05:39.604604 2309 raft_consensus.cc:2454] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0ec74dff413420bbd1e9164226ae5c6 in term 1.
I20260620 03:05:39.605018 2515 leader_election.cc:302] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2882899786974c498b4f4c43442563bd, b0ec74dff413420bbd1e9164226ae5c6; no voters:
I20260620 03:05:39.605538 2685 raft_consensus.cc:2781] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:05:39.605722 2685 raft_consensus.cc:686] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [term 1 LEADER]: Becoming Leader. State: Replica: b0ec74dff413420bbd1e9164226ae5c6, State: Running, Role: LEADER
I20260620 03:05:39.606303 2685 consensus_queue.cc:237] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:39.609402 2147 catalog_manager.cc:5495] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0ec74dff413420bbd1e9164226ae5c6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b0ec74dff413420bbd1e9164226ae5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } health_report { overall_health: HEALTHY } } }
I20260620 03:05:39.647114 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:39.640744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34346 (local address 127.0.97.3:46439)
0620 03:05:39.640916 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:39.640924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:39.641149 (+ 225us) server_negotiation.cc:407] Connection header received
0620 03:05:39.641340 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:39.641346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:39.641426 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:39.641521 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:05:39.641930 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.643082 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.645056 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.645767 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.645896 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:39.646320 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:05:39.646474 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:05:39.646562 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:39.646721 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 03:05:39.646795 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 03:05:39.662395 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:39.656243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44131 (local address 127.0.97.3:46439)
0620 03:05:39.656409 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:39.656416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:39.656644 (+ 228us) server_negotiation.cc:407] Connection header received
0620 03:05:39.656804 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:39.656812 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:39.656892 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:39.656989 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:05:39.657325 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.658502 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.660442 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.661079 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.661234 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:39.661654 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:05:39.661775 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:05:39.661865 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:39.662001 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 03:05:39.662074 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":49}
I20260620 03:05:39.662425 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:39.656293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46439 (local address 127.0.97.3:44131)
0620 03:05:39.656575 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:39.656593 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:39.656691 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:39.657015 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:39.657019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:39.657030 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 03:05:39.657198 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:39.657206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.658648 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.658655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:39.660301 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:39.660314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:39.661230 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:39.661236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:39.661449 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:39.661493 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 03:05:39.661878 (+ 385us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 03:05:39.661888 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 03:05:39.661962 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 03:05:39.662090 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":4}
I20260620 03:05:39.673792 2309 raft_consensus.cc:1261] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0 [term 1 FOLLOWER]: Refusing update from remote peer b0ec74dff413420bbd1e9164226ae5c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:05:39.673806 2446 raft_consensus.cc:1261] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd [term 1 FOLLOWER]: Refusing update from remote peer b0ec74dff413420bbd1e9164226ae5c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:05:39.674394 2675 consensus_queue.cc:1035] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:05:39.674849 2675 consensus_queue.cc:1035] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:05:39.693281 2680 mvcc.cc:205] Tried to move back new op lower bound from 7298763733698617344 to 7298763733430329344. Current Snapshot: MvccSnapshot[applied={T|T < 7298763733698617344}]
W20260620 03:05:39.717345 2680 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 03:05:41.322261 2147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:05:41.322497 2147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:05:41.326253 2147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e: Sending DeleteTablet for 3 replicas of tablet ae32906ff8af493897a1c5683a6f3657
I20260620 03:05:41.326957 2284 tablet_service.cc:1514] Processing DeleteTablet for tablet ae32906ff8af493897a1c5683a6f3657 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:05:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41318
I20260620 03:05:41.327056 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet ae32906ff8af493897a1c5683a6f3657 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:05:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34328
I20260620 03:05:41.327169 2423 tablet_service.cc:1514] Processing DeleteTablet for tablet ae32906ff8af493897a1c5683a6f3657 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:05:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60844
I20260620 03:05:41.337426 2732 tablet_replica.cc:330] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0: stopping tablet replica
I20260620 03:05:41.337621 2734 tablet_replica.cc:330] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd: stopping tablet replica
I20260620 03:05:41.337740 2732 raft_consensus.cc:2229] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:05:41.337854 2734 raft_consensus.cc:2229] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:05:41.337890 2733 tablet_replica.cc:330] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6: stopping tablet replica
I20260620 03:05:41.338006 2732 raft_consensus.cc:2258] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:05:41.338096 2733 raft_consensus.cc:2229] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:05:41.338119 2734 raft_consensus.cc:2258] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:05:41.338656 2733 raft_consensus.cc:2258] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:05:41.339473 2732 ts_tablet_manager.cc:1905] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:05:41.339586 2734 ts_tablet_manager.cc:1905] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:05:41.340081 2733 ts_tablet_manager.cc:1905] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:05:41.346066 2732 ts_tablet_manager.cc:1918] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 03:05:41.346276 2732 log.cc:1196] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/ae32906ff8af493897a1c5683a6f3657
I20260620 03:05:41.346478 2733 ts_tablet_manager.cc:1918] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 03:05:41.346647 2733 log.cc:1196] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/ae32906ff8af493897a1c5683a6f3657
I20260620 03:05:41.346829 2732 ts_tablet_manager.cc:1939] T ae32906ff8af493897a1c5683a6f3657 P c60ce80299644581886852102ca91fa0: Deleting consensus metadata
I20260620 03:05:41.347065 2734 ts_tablet_manager.cc:1918] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 03:05:41.347138 2733 ts_tablet_manager.cc:1939] T ae32906ff8af493897a1c5683a6f3657 P b0ec74dff413420bbd1e9164226ae5c6: Deleting consensus metadata
I20260620 03:05:41.347258 2734 log.cc:1196] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/ae32906ff8af493897a1c5683a6f3657
I20260620 03:05:41.347767 2734 ts_tablet_manager.cc:1939] T ae32906ff8af493897a1c5683a6f3657 P 2882899786974c498b4f4c43442563bd: Deleting consensus metadata
I20260620 03:05:41.348111 2133 catalog_manager.cc:4841] TS c60ce80299644581886852102ca91fa0 (127.0.97.1:44911): tablet ae32906ff8af493897a1c5683a6f3657 (table test-table [id=fe22d826eaca481592dfd89fa7031b8c]) successfully deleted
I20260620 03:05:41.348202 2132 catalog_manager.cc:4841] TS b0ec74dff413420bbd1e9164226ae5c6 (127.0.97.3:46439): tablet ae32906ff8af493897a1c5683a6f3657 (table test-table [id=fe22d826eaca481592dfd89fa7031b8c]) successfully deleted
I20260620 03:05:41.348949 2133 catalog_manager.cc:4841] TS 2882899786974c498b4f4c43442563bd (127.0.97.2:38757): tablet ae32906ff8af493897a1c5683a6f3657 (table test-table [id=fe22d826eaca481592dfd89fa7031b8c]) successfully deleted
Jun 20 03:05:41 dist-test-slave-xv68 krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924741, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 03:05:41 dist-test-slave-xv68 krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924741, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:41.381173 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:41.367012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.0.97.62:45311)
0620 03:05:41.367579 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:41.367587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:41.367612 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:05:41.367830 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:41.367834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:41.367893 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:41.368001 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:41.369219 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.369991 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.371205 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.371666 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.371804 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:41.376167 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:41.376172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:41.376175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:41.376207 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:41.378577 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:41.379227 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:41.379234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:41.379238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:41.379415 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:41.379772 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:41.379778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:41.379781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:41.380056 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:41.380304 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:41.380609 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 03:05:41.380828 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":303,"threads_started":1}
I20260620 03:05:41.384368 2147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:05:41.384986 2147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:05:41.390961 2423 tablet_service.cc:1467] Processing CreateTablet for tablet d4dedc0c13e44b42b3df7219cc10e144 (DEFAULT_TABLE table=test-table [id=9f9881f7a20e4c5593eead18799f4066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:41.390978 2284 tablet_service.cc:1467] Processing CreateTablet for tablet d4dedc0c13e44b42b3df7219cc10e144 (DEFAULT_TABLE table=test-table [id=9f9881f7a20e4c5593eead18799f4066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:41.391129 2562 tablet_service.cc:1467] Processing CreateTablet for tablet d4dedc0c13e44b42b3df7219cc10e144 (DEFAULT_TABLE table=test-table [id=9f9881f7a20e4c5593eead18799f4066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:41.391403 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4dedc0c13e44b42b3df7219cc10e144. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:41.391403 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4dedc0c13e44b42b3df7219cc10e144. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:41.391580 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4dedc0c13e44b42b3df7219cc10e144. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:41.398737 2743 tablet_bootstrap.cc:492] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6: Bootstrap starting.
I20260620 03:05:41.402211 2743 tablet_bootstrap.cc:654] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:41.403677 2744 tablet_bootstrap.cc:492] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd: Bootstrap starting.
I20260620 03:05:41.404886 2743 tablet_bootstrap.cc:492] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6: No bootstrap required, opened a new log
I20260620 03:05:41.405104 2743 ts_tablet_manager.cc:1397] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 03:05:41.405450 2744 tablet_bootstrap.cc:654] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:41.405762 2745 tablet_bootstrap.cc:492] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0: Bootstrap starting.
I20260620 03:05:41.405999 2743 raft_consensus.cc:348] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:41.406242 2743 raft_consensus.cc:374] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:41.406359 2743 raft_consensus.cc:729] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ec74dff413420bbd1e9164226ae5c6, State: Initialized, Role: FOLLOWER
I20260620 03:05:41.406594 2743 consensus_queue.cc:260] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:41.407394 2743 ts_tablet_manager.cc:1428] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 03:05:41.408529 2745 tablet_bootstrap.cc:654] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:41.409162 2744 tablet_bootstrap.cc:492] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd: No bootstrap required, opened a new log
I20260620 03:05:41.409296 2744 ts_tablet_manager.cc:1397] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 03:05:41.410060 2744 raft_consensus.cc:348] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:41.410306 2744 raft_consensus.cc:374] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:41.410432 2744 raft_consensus.cc:729] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2882899786974c498b4f4c43442563bd, State: Initialized, Role: FOLLOWER
I20260620 03:05:41.410697 2744 consensus_queue.cc:260] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:41.411469 2744 ts_tablet_manager.cc:1428] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 03:05:41.411708 2745 tablet_bootstrap.cc:492] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0: No bootstrap required, opened a new log
I20260620 03:05:41.411823 2745 ts_tablet_manager.cc:1397] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 03:05:41.412437 2745 raft_consensus.cc:348] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:41.412606 2745 raft_consensus.cc:374] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:41.412708 2745 raft_consensus.cc:729] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60ce80299644581886852102ca91fa0, State: Initialized, Role: FOLLOWER
I20260620 03:05:41.412956 2745 consensus_queue.cc:260] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:41.413678 2745 ts_tablet_manager.cc:1428] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 03:05:41.465945 2749 raft_consensus.cc:482] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:05:41.466193 2749 raft_consensus.cc:504] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:41.467636 2749 leader_election.cc:288] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c60ce80299644581886852102ca91fa0 (127.0.97.1:44911), b0ec74dff413420bbd1e9164226ae5c6 (127.0.97.3:46439)
I20260620 03:05:41.485392 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:41.468024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60885 (local address 127.0.97.1:44911)
0620 03:05:41.479626 (+ 11602us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:41.479632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:41.479658 (+ 26us) server_negotiation.cc:407] Connection header received
0620 03:05:41.479738 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:41.479742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:41.479799 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:41.479931 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:41.480990 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.481757 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.483613 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.484365 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.484511 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:41.484620 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:41.484936 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 03:05:41.485063 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11459,"thread_start_us":11279,"threads_started":1}
I20260620 03:05:41.485440 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:41.467923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44911 (local address 127.0.97.2:60885)
0620 03:05:41.479158 (+ 11235us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:41.479184 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:41.479376 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:41.479953 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:41.479957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:41.479983 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:41.480267 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:41.480275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.481906 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.481911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:41.483461 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:41.483473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.484566 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.484572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:41.484745 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:41.484772 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 03:05:41.484906 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 03:05:41.485036 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11065,"mutex_wait_us":64,"thread_start_us":11604,"threads_started":1}
I20260620 03:05:41.485579 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:41.468022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56699 (local address 127.0.97.3:46439)
0620 03:05:41.478915 (+ 10893us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:41.478921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:41.479900 (+ 979us) server_negotiation.cc:407] Connection header received
0620 03:05:41.480091 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:41.480096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:41.480157 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:41.480259 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:41.481270 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.482033 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.483820 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.484570 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.484736 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:41.484845 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:41.485118 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 03:05:41.485247 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10713,"thread_start_us":10565,"threads_started":1}
I20260620 03:05:41.485841 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:41.467923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46439 (local address 127.0.97.2:56699)
0620 03:05:41.479751 (+ 11828us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:41.479765 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:41.479957 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:41.480282 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:41.480286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:41.480310 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:41.481091 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:41.481101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.482170 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.482175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:41.483626 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:41.483637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.484794 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.484800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:41.484957 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:41.484979 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 03:05:41.485082 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 03:05:41.485208 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11727,"mutex_wait_us":34,"thread_start_us":10882,"threads_started":1}
I20260620 03:05:41.486140 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dedc0c13e44b42b3df7219cc10e144" candidate_uuid: "2882899786974c498b4f4c43442563bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60ce80299644581886852102ca91fa0" is_pre_election: true
I20260620 03:05:41.486395 2308 raft_consensus.cc:2454] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2882899786974c498b4f4c43442563bd in term 0.
I20260620 03:05:41.486505 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dedc0c13e44b42b3df7219cc10e144" candidate_uuid: "2882899786974c498b4f4c43442563bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ec74dff413420bbd1e9164226ae5c6" is_pre_election: true
I20260620 03:05:41.486723 2587 raft_consensus.cc:2454] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2882899786974c498b4f4c43442563bd in term 0.
I20260620 03:05:41.486979 2376 leader_election.cc:302] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2882899786974c498b4f4c43442563bd, c60ce80299644581886852102ca91fa0; no voters:
I20260620 03:05:41.487469 2749 raft_consensus.cc:2781] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:05:41.487598 2749 raft_consensus.cc:482] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:05:41.487689 2749 raft_consensus.cc:3037] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:41.489800 2749 raft_consensus.cc:504] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:41.490419 2749 leader_election.cc:288] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [CANDIDATE]: Term 1 election: Requested vote from peers c60ce80299644581886852102ca91fa0 (127.0.97.1:44911), b0ec74dff413420bbd1e9164226ae5c6 (127.0.97.3:46439)
I20260620 03:05:41.490768 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dedc0c13e44b42b3df7219cc10e144" candidate_uuid: "2882899786974c498b4f4c43442563bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60ce80299644581886852102ca91fa0"
I20260620 03:05:41.490970 2308 raft_consensus.cc:3037] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:41.490967 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dedc0c13e44b42b3df7219cc10e144" candidate_uuid: "2882899786974c498b4f4c43442563bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ec74dff413420bbd1e9164226ae5c6"
I20260620 03:05:41.491202 2587 raft_consensus.cc:3037] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:41.493569 2308 raft_consensus.cc:2454] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2882899786974c498b4f4c43442563bd in term 1.
I20260620 03:05:41.493646 2587 raft_consensus.cc:2454] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2882899786974c498b4f4c43442563bd in term 1.
I20260620 03:05:41.494100 2375 leader_election.cc:302] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2882899786974c498b4f4c43442563bd, b0ec74dff413420bbd1e9164226ae5c6, c60ce80299644581886852102ca91fa0; no voters:
I20260620 03:05:41.494375 2749 raft_consensus.cc:2781] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:05:41.494838 2749 raft_consensus.cc:686] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [term 1 LEADER]: Becoming Leader. State: Replica: 2882899786974c498b4f4c43442563bd, State: Running, Role: LEADER
I20260620 03:05:41.495198 2749 consensus_queue.cc:237] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:41.498533 2146 catalog_manager.cc:5495] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2882899786974c498b4f4c43442563bd (127.0.97.2). New cstate: current_term: 1 leader_uuid: "2882899786974c498b4f4c43442563bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } health_report { overall_health: UNKNOWN } } }
I20260620 03:05:41.533253 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:41.526033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60860 (local address 127.0.97.2:38757)
0620 03:05:41.527681 (+ 1648us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:41.527685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:41.527702 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:05:41.527780 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:41.527783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:41.527851 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:41.528032 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:05:41.528429 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.529228 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.530560 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.530982 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.531118 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:41.531595 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:05:41.532195 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:05:41.532363 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:41.532433 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 03:05:41.532507 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1547,"thread_start_us":1438,"threads_started":1}
I20260620 03:05:41.546799 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:41.542267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54565 (local address 127.0.97.3:46439)
0620 03:05:41.542495 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:41.542502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:41.542836 (+ 334us) server_negotiation.cc:407] Connection header received
0620 03:05:41.542986 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:41.542990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:41.543051 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:41.543124 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:05:41.543584 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.544242 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.545468 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.545866 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.545967 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:41.546310 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:05:41.546395 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:05:41.546460 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:41.546567 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 03:05:41.546621 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 03:05:41.546881 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:41.542154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46439 (local address 127.0.97.2:54565)
0620 03:05:41.542746 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:41.542761 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:41.542869 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:41.543151 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:41.543154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:41.543162 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 03:05:41.543467 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:41.543475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.544347 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.544350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:41.545360 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:41.545370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.546008 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.546012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:41.546130 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:41.546175 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 03:05:41.546475 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 03:05:41.546482 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 03:05:41.546539 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 03:05:41.546625 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":319,"threads_started":1}
I20260620 03:05:41.554103 2587 raft_consensus.cc:1261] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6 [term 1 FOLLOWER]: Refusing update from remote peer 2882899786974c498b4f4c43442563bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:05:41.554209 2308 raft_consensus.cc:1261] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0 [term 1 FOLLOWER]: Refusing update from remote peer 2882899786974c498b4f4c43442563bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:05:41.554849 2755 consensus_queue.cc:1035] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:05:41.555207 2749 consensus_queue.cc:1035] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:05:41.581424 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:41.574966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38997 (local address 127.0.97.2:38757)
0620 03:05:41.575178 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:41.575187 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:41.575879 (+ 692us) server_negotiation.cc:407] Connection header received
0620 03:05:41.575975 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:41.575979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:41.576034 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:41.576231 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:05:41.576677 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.577404 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.579757 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.580219 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.580316 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:41.580696 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:05:41.580792 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:05:41.580888 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:41.581028 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 03:05:41.581125 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 03:05:41.581507 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:41.574814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38757 (local address 127.0.97.3:38997)
0620 03:05:41.575723 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:41.575739 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:41.575845 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:41.576247 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:41.576252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:41.576263 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 03:05:41.576539 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:41.576547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.577546 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.577552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:41.579598 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:41.579616 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.580367 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.580371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:41.580495 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:41.580529 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 03:05:41.580887 (+ 358us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 03:05:41.580900 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 03:05:41.580999 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 03:05:41.581130 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":673,"threads_started":1}
I20260620 03:05:41.612401 2146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:05:41.612689 2146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:05:41.615789 2146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e: Sending DeleteTablet for 3 replicas of tablet d4dedc0c13e44b42b3df7219cc10e144
I20260620 03:05:41.616559 2423 tablet_service.cc:1514] Processing DeleteTablet for tablet d4dedc0c13e44b42b3df7219cc10e144 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:05:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60844
I20260620 03:05:41.616701 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet d4dedc0c13e44b42b3df7219cc10e144 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:05:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34328
I20260620 03:05:41.616869 2284 tablet_service.cc:1514] Processing DeleteTablet for tablet d4dedc0c13e44b42b3df7219cc10e144 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:05:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41318
I20260620 03:05:41.617132 2732 tablet_replica.cc:330] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0: stopping tablet replica
I20260620 03:05:41.617341 2732 raft_consensus.cc:2229] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:05:41.617532 2732 raft_consensus.cc:2258] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:05:41.617817 2733 tablet_replica.cc:330] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6: stopping tablet replica
I20260620 03:05:41.618013 2733 raft_consensus.cc:2229] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:05:41.618206 2733 raft_consensus.cc:2258] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:05:41.618475 2734 tablet_replica.cc:330] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd: stopping tablet replica
I20260620 03:05:41.618578 2732 ts_tablet_manager.cc:1905] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:05:41.618690 2734 raft_consensus.cc:2229] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:05:41.619197 2734 raft_consensus.cc:2258] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:05:41.619326 2733 ts_tablet_manager.cc:1905] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:05:41.621181 2734 ts_tablet_manager.cc:1905] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:05:41.623589 2146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:05:41.624238 2146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:05:41.624889 2732 ts_tablet_manager.cc:1918] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 03:05:41.625420 2733 ts_tablet_manager.cc:1918] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 03:05:41.625476 2732 log.cc:1196] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/d4dedc0c13e44b42b3df7219cc10e144
I20260620 03:05:41.625568 2733 log.cc:1196] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/d4dedc0c13e44b42b3df7219cc10e144
I20260620 03:05:41.625926 2732 ts_tablet_manager.cc:1939] T d4dedc0c13e44b42b3df7219cc10e144 P c60ce80299644581886852102ca91fa0: Deleting consensus metadata
I20260620 03:05:41.626968 2133 catalog_manager.cc:4841] TS c60ce80299644581886852102ca91fa0 (127.0.97.1:44911): tablet d4dedc0c13e44b42b3df7219cc10e144 (table test-table [id=9f9881f7a20e4c5593eead18799f4066]) successfully deleted
I20260620 03:05:41.628489 2733 ts_tablet_manager.cc:1939] T d4dedc0c13e44b42b3df7219cc10e144 P b0ec74dff413420bbd1e9164226ae5c6: Deleting consensus metadata
I20260620 03:05:41.628927 2734 ts_tablet_manager.cc:1918] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 03:05:41.629153 2734 log.cc:1196] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/d4dedc0c13e44b42b3df7219cc10e144
I20260620 03:05:41.629765 2734 ts_tablet_manager.cc:1939] T d4dedc0c13e44b42b3df7219cc10e144 P 2882899786974c498b4f4c43442563bd: Deleting consensus metadata
I20260620 03:05:41.630686 2132 catalog_manager.cc:4841] TS b0ec74dff413420bbd1e9164226ae5c6 (127.0.97.3:46439): tablet d4dedc0c13e44b42b3df7219cc10e144 (table test-table [id=9f9881f7a20e4c5593eead18799f4066]) successfully deleted
I20260620 03:05:41.630790 2133 catalog_manager.cc:4841] TS 2882899786974c498b4f4c43442563bd (127.0.97.2:38757): tablet d4dedc0c13e44b42b3df7219cc10e144 (table test-table [id=9f9881f7a20e4c5593eead18799f4066]) successfully deleted
I20260620 03:05:41.631875 2284 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6070f4868f4e4997427ed1b377887b (DEFAULT_TABLE table=test-table [id=53816e37775841daa5a0476006bf92c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:41.632288 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6070f4868f4e4997427ed1b377887b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:41.632411 2423 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6070f4868f4e4997427ed1b377887b (DEFAULT_TABLE table=test-table [id=53816e37775841daa5a0476006bf92c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:41.632769 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6070f4868f4e4997427ed1b377887b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:41.632833 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6070f4868f4e4997427ed1b377887b (DEFAULT_TABLE table=test-table [id=53816e37775841daa5a0476006bf92c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:41.633208 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6070f4868f4e4997427ed1b377887b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:41.638821 2745 tablet_bootstrap.cc:492] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0: Bootstrap starting.
I20260620 03:05:41.640403 2745 tablet_bootstrap.cc:654] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:41.641397 2743 tablet_bootstrap.cc:492] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6: Bootstrap starting.
I20260620 03:05:41.641839 2744 tablet_bootstrap.cc:492] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd: Bootstrap starting.
I20260620 03:05:41.642755 2743 tablet_bootstrap.cc:654] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:41.643931 2744 tablet_bootstrap.cc:654] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:41.648615 2745 tablet_bootstrap.cc:492] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0: No bootstrap required, opened a new log
I20260620 03:05:41.649228 2745 ts_tablet_manager.cc:1397] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20260620 03:05:41.650338 2745 raft_consensus.cc:348] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:41.651902 2745 raft_consensus.cc:374] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:41.652071 2745 raft_consensus.cc:729] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60ce80299644581886852102ca91fa0, State: Initialized, Role: FOLLOWER
I20260620 03:05:41.652489 2745 consensus_queue.cc:260] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:41.653292 2745 ts_tablet_manager.cc:1428] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260620 03:05:41.655201 2744 tablet_bootstrap.cc:492] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd: No bootstrap required, opened a new log
I20260620 03:05:41.655429 2744 ts_tablet_manager.cc:1397] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
I20260620 03:05:41.655920 2743 tablet_bootstrap.cc:492] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6: No bootstrap required, opened a new log
I20260620 03:05:41.656077 2743 ts_tablet_manager.cc:1397] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.003s
I20260620 03:05:41.656198 2744 raft_consensus.cc:348] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:41.656447 2744 raft_consensus.cc:374] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:41.656527 2744 raft_consensus.cc:729] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2882899786974c498b4f4c43442563bd, State: Initialized, Role: FOLLOWER
I20260620 03:05:41.656780 2744 consensus_queue.cc:260] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:41.658001 2744 ts_tablet_manager.cc:1428] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 03:05:41.657966 2743 raft_consensus.cc:348] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:41.658205 2743 raft_consensus.cc:374] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:41.658303 2743 raft_consensus.cc:729] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ec74dff413420bbd1e9164226ae5c6, State: Initialized, Role: FOLLOWER
I20260620 03:05:41.658538 2743 consensus_queue.cc:260] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:41.659636 2743 ts_tablet_manager.cc:1428] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 03:05:41.738159 2685 raft_consensus.cc:482] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:05:41.738420 2685 raft_consensus.cc:504] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:41.739004 2685 leader_election.cc:288] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c60ce80299644581886852102ca91fa0 (127.0.97.1:44911), 2882899786974c498b4f4c43442563bd (127.0.97.2:38757)
I20260620 03:05:41.740152 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6070f4868f4e4997427ed1b377887b" candidate_uuid: "b0ec74dff413420bbd1e9164226ae5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60ce80299644581886852102ca91fa0" is_pre_election: true
I20260620 03:05:41.740155 2447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6070f4868f4e4997427ed1b377887b" candidate_uuid: "b0ec74dff413420bbd1e9164226ae5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2882899786974c498b4f4c43442563bd" is_pre_election: true
I20260620 03:05:41.740401 2308 raft_consensus.cc:2454] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0ec74dff413420bbd1e9164226ae5c6 in term 0.
I20260620 03:05:41.740409 2447 raft_consensus.cc:2454] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0ec74dff413420bbd1e9164226ae5c6 in term 0.
I20260620 03:05:41.740824 2515 leader_election.cc:302] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0ec74dff413420bbd1e9164226ae5c6, c60ce80299644581886852102ca91fa0; no voters:
I20260620 03:05:41.741115 2685 raft_consensus.cc:2781] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:05:41.741251 2685 raft_consensus.cc:482] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:05:41.741377 2685 raft_consensus.cc:3037] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:41.743274 2685 raft_consensus.cc:504] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:41.743741 2685 leader_election.cc:288] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [CANDIDATE]: Term 1 election: Requested vote from peers c60ce80299644581886852102ca91fa0 (127.0.97.1:44911), 2882899786974c498b4f4c43442563bd (127.0.97.2:38757)
I20260620 03:05:41.744212 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6070f4868f4e4997427ed1b377887b" candidate_uuid: "b0ec74dff413420bbd1e9164226ae5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60ce80299644581886852102ca91fa0"
I20260620 03:05:41.744290 2447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6070f4868f4e4997427ed1b377887b" candidate_uuid: "b0ec74dff413420bbd1e9164226ae5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2882899786974c498b4f4c43442563bd"
I20260620 03:05:41.744428 2308 raft_consensus.cc:3037] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:41.744472 2447 raft_consensus.cc:3037] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:41.747423 2308 raft_consensus.cc:2454] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0ec74dff413420bbd1e9164226ae5c6 in term 1.
I20260620 03:05:41.747480 2447 raft_consensus.cc:2454] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0ec74dff413420bbd1e9164226ae5c6 in term 1.
I20260620 03:05:41.747822 2515 leader_election.cc:302] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0ec74dff413420bbd1e9164226ae5c6, c60ce80299644581886852102ca91fa0; no voters:
I20260620 03:05:41.748078 2685 raft_consensus.cc:2781] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:05:41.748243 2685 raft_consensus.cc:686] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [term 1 LEADER]: Becoming Leader. State: Replica: b0ec74dff413420bbd1e9164226ae5c6, State: Running, Role: LEADER
I20260620 03:05:41.748512 2685 consensus_queue.cc:237] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } }
I20260620 03:05:41.751708 2146 catalog_manager.cc:5495] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0ec74dff413420bbd1e9164226ae5c6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b0ec74dff413420bbd1e9164226ae5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0ec74dff413420bbd1e9164226ae5c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46439 } health_report { overall_health: HEALTHY } } }
I20260620 03:05:41.783262 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:41.777623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.0.97.3:46439)
0620 03:05:41.777774 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:41.777779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:41.778092 (+ 313us) server_negotiation.cc:407] Connection header received
0620 03:05:41.778276 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:41.778279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:41.778339 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:41.778425 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:05:41.778810 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.779582 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.781446 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:41.782038 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:41.782138 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:41.782581 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:05:41.782673 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:05:41.782737 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:41.782886 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 03:05:41.782954 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 03:05:41.792675 2308 raft_consensus.cc:1261] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0 [term 1 FOLLOWER]: Refusing update from remote peer b0ec74dff413420bbd1e9164226ae5c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:05:41.793289 2727 consensus_queue.cc:1035] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60ce80299644581886852102ca91fa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:05:41.793511 2447 raft_consensus.cc:1261] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd [term 1 FOLLOWER]: Refusing update from remote peer b0ec74dff413420bbd1e9164226ae5c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:05:41.794100 2727 consensus_queue.cc:1035] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2882899786974c498b4f4c43442563bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 03:05:41.814836 2712 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 03:05:43.397904 2146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:05:43.398180 2146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:05:43.403551 2146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07ce8814fbb84557a1053a589b6bd10e: Sending DeleteTablet for 3 replicas of tablet 4a6070f4868f4e4997427ed1b377887b
I20260620 03:05:43.404311 2284 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6070f4868f4e4997427ed1b377887b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:05:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41318
I20260620 03:05:43.404361 2423 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6070f4868f4e4997427ed1b377887b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:05:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60844
I20260620 03:05:43.404521 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6070f4868f4e4997427ed1b377887b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:05:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34328
I20260620 03:05:43.405212 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 2218
I20260620 03:05:43.415326 2795 tablet_replica.cc:330] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd: stopping tablet replica
I20260620 03:05:43.416179 2795 raft_consensus.cc:2229] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:05:43.416762 2795 raft_consensus.cc:2258] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:05:43.417183 2794 tablet_replica.cc:330] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0: stopping tablet replica
I20260620 03:05:43.417510 2794 raft_consensus.cc:2229] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:05:43.417873 2794 raft_consensus.cc:2258] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:05:43.418438 2795 ts_tablet_manager.cc:1905] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:05:43.419083 2794 ts_tablet_manager.cc:1905] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:05:43.425202 2795 ts_tablet_manager.cc:1918] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 03:05:43.425364 2795 log.cc:1196] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/4a6070f4868f4e4997427ed1b377887b
I20260620 03:05:43.425734 2795 ts_tablet_manager.cc:1939] T 4a6070f4868f4e4997427ed1b377887b P 2882899786974c498b4f4c43442563bd: Deleting consensus metadata
I20260620 03:05:43.426906 2133 catalog_manager.cc:4841] TS 2882899786974c498b4f4c43442563bd (127.0.97.2:38757): tablet 4a6070f4868f4e4997427ed1b377887b (table test-table [id=53816e37775841daa5a0476006bf92c5]) successfully deleted
I20260620 03:05:43.427075 2796 tablet_replica.cc:330] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6: stopping tablet replica
I20260620 03:05:43.427284 2796 raft_consensus.cc:2229] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:05:43.427708 2796 raft_consensus.cc:2258] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:05:43.427937 2794 ts_tablet_manager.cc:1918] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 03:05:43.428114 2794 log.cc:1196] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/4a6070f4868f4e4997427ed1b377887b
I20260620 03:05:43.428536 2794 ts_tablet_manager.cc:1939] T 4a6070f4868f4e4997427ed1b377887b P c60ce80299644581886852102ca91fa0: Deleting consensus metadata
I20260620 03:05:43.428624 2796 ts_tablet_manager.cc:1905] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:05:43.429522 2133 catalog_manager.cc:4841] TS c60ce80299644581886852102ca91fa0 (127.0.97.1:44911): tablet 4a6070f4868f4e4997427ed1b377887b (table test-table [id=53816e37775841daa5a0476006bf92c5]) successfully deleted
Jun 20 03:05:43 dist-test-slave-xv68 krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924741, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:05:43.434770 2796 ts_tablet_manager.cc:1918] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 03:05:43.434917 2796 log.cc:1196] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/4a6070f4868f4e4997427ed1b377887b
I20260620 03:05:43.435416 2796 ts_tablet_manager.cc:1939] T 4a6070f4868f4e4997427ed1b377887b P b0ec74dff413420bbd1e9164226ae5c6: Deleting consensus metadata
I20260620 03:05:43.436446 2132 catalog_manager.cc:4841] TS b0ec74dff413420bbd1e9164226ae5c6 (127.0.97.3:46439): tablet 4a6070f4868f4e4997427ed1b377887b (table test-table [id=53816e37775841daa5a0476006bf92c5]) successfully deleted
I20260620 03:05:43.438759 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:43.408302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41328 (local address 127.0.97.1:44911)
0620 03:05:43.415926 (+ 7624us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:43.415936 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:43.416524 (+ 588us) server_negotiation.cc:407] Connection header received
0620 03:05:43.416744 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:43.416756 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:43.416861 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:43.416983 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:43.418845 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:43.420048 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:43.421934 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:43.422555 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:43.422688 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:43.432684 (+ 9996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:43.432706 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:43.432729 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:43.432774 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:43.436434 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:43.437099 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:43.437107 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:43.437110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:43.437235 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:43.437570 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:43.437573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:43.437575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:43.437918 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:43.438134 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:43.438351 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 03:05:43.438510 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7382,"thread_start_us":7204,"threads_started":1}
W20260620 03:05:43.439529 2349 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41328
W20260620 03:05:43.440104 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 2218: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:05:43.440308 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 2218
I20260620 03:05:43.462625 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 2358
Jun 20 03:05:43 dist-test-slave-xv68 krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924741, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:05:43.477504 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:43.463359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60868 (local address 127.0.97.2:38757)
0620 03:05:43.463890 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:43.463894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:43.463910 (+ 16us) server_negotiation.cc:407] Connection header received
0620 03:05:43.463961 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:43.463965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:43.464018 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:43.464110 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:43.465266 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:43.465984 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:43.467187 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:43.467699 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:43.467815 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:43.471949 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:43.471962 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:43.471978 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:43.472009 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:43.474670 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:43.475459 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:43.475466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:43.475469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:43.475614 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:43.475991 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:43.475999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:43.476002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:43.476326 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:43.476638 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:43.476927 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 03:05:43.477162 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":316,"threads_started":1}
W20260620 03:05:43.478168 2488 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60868
W20260620 03:05:43.478832 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 2358: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:05:43.479022 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 2358
I20260620 03:05:43.501245 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 2497
Jun 20 03:05:43 dist-test-slave-xv68 krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924741, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:05:43.516688 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:43.502066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.0.97.3:46439)
0620 03:05:43.502643 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:43.502647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:43.502661 (+ 14us) server_negotiation.cc:407] Connection header received
0620 03:05:43.502716 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:43.502720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:43.502775 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:43.502894 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:43.504296 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:43.505114 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:43.506385 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:43.506788 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:43.506918 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:43.511120 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:43.511135 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:43.511152 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:43.511188 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:43.513895 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:43.514560 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:43.514568 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:43.514571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:43.514704 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:43.515085 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:43.515092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:43.515095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:43.515512 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:43.515791 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:43.516109 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 03:05:43.516357 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":270,"threads_started":1}
W20260620 03:05:43.517386 2627 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34358
W20260620 03:05:43.517998 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 2497: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:05:43.518173 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 2497
I20260620 03:05:43.542768 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 2115
I20260620 03:05:43.544088 2187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:05:43.659704 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 2115
2026-06-20T03:05:43Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8309 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:05:43 dist-test-slave-xv68 krb5kdc[2804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:05:43 dist-test-slave-xv68 krb5kdc[2804](info): set up 2 sockets
Jun 20 03:05:43 dist-test-slave-xv68 krb5kdc[2804](info): commencing operation
krb5kdc: starting...
W20260620 03:05:44.739480 388 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:05:44 dist-test-slave-xv68 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:05:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:05:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:05:44.924973 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38813
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:45.254079 2817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:45.254369 2817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:45.254523 2817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:45.261888 2817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:05:45.261998 2817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:45.262077 2817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:45.262162 2817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:05:45.262243 2817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:05:45.272190 2817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:45.273541 2817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:45.274940 2817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:45.287061 2822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:45.287102 2825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:45.287174 2823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:45.287434 2817 server_base.cc:1034] running on GCE node
I20260620 03:05:45.288515 2817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:45.289912 2817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:45.291160 2817 hybrid_clock.cc:648] HybridClock initialized: now 1781924745291107 us; error 64 us; skew 500 ppm
I20260620 03:05:45.291558 2817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:45 dist-test-slave-xv68 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:45.294741 2817 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:05:45.296309 2817 webserver.cc:460] Webserver started at http://127.0.97.62:36101/ using document root <none> and password file <none>
I20260620 03:05:45.297008 2817 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:45.297152 2817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:45.297466 2817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:45.300431 2817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "5d204e6da3f941059733a229397f435d"
format_stamp: "Formatted at 2026-06-20 03:05:45 on dist-test-slave-xv68"
server_key: "7900852f947ee3b5f74932b4c6b2794a"
server_key_iv: "c8a1346eb5ec6995ea286c4f5283139b"
server_key_version: "encryptionkey@0"
I20260620 03:05:45.301187 2817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "5d204e6da3f941059733a229397f435d"
format_stamp: "Formatted at 2026-06-20 03:05:45 on dist-test-slave-xv68"
server_key: "7900852f947ee3b5f74932b4c6b2794a"
server_key_iv: "c8a1346eb5ec6995ea286c4f5283139b"
server_key_version: "encryptionkey@0"
I20260620 03:05:45.306139 2817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:05:45.309461 2832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:45.310801 2817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 03:05:45.311035 2817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "5d204e6da3f941059733a229397f435d"
format_stamp: "Formatted at 2026-06-20 03:05:45 on dist-test-slave-xv68"
server_key: "7900852f947ee3b5f74932b4c6b2794a"
server_key_iv: "c8a1346eb5ec6995ea286c4f5283139b"
server_key_version: "encryptionkey@0"
I20260620 03:05:45.311235 2817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:45.345328 2817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:45.349431 2817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:45.349752 2817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:45.367514 2817 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38813
I20260620 03:05:45.367550 2883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38813 every 8 connection(s)
I20260620 03:05:45.368842 2817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:05:45.372437 2884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:45.377913 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 2817
I20260620 03:05:45.378168 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:05:45.378786 388 external_mini_cluster.cc:1428] Setting key 532aaf05be54c99fdd63189eec985360
I20260620 03:05:45.383059 2884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d: Bootstrap starting.
I20260620 03:05:45.386845 2884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:45.388041 2884 log.cc:824] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:45.391147 2884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d: No bootstrap required, opened a new log
Jun 20 03:05:45 dist-test-slave-xv68 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:45.402488 2884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d204e6da3f941059733a229397f435d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38813 } }
I20260620 03:05:45.403013 2884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:45.403131 2884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d204e6da3f941059733a229397f435d, State: Initialized, Role: FOLLOWER
I20260620 03:05:45.403898 2884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d204e6da3f941059733a229397f435d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38813 } }
I20260620 03:05:45.404210 2884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:05:45.404336 2884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:05:45.404546 2884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:45.406061 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:45.380856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58472 (local address 127.0.97.62:38813)
0620 03:05:45.381651 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:45.381664 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:45.381704 (+ 40us) server_negotiation.cc:407] Connection header received
0620 03:05:45.383568 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:45.383591 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:45.383995 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:45.384328 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:45.386172 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:45.387680 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:45.389240 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:45.389926 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:45.390058 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:45.395068 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:45.395079 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:45.395100 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:45.395160 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:45.400025 (+ 4865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:45.400959 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:45.400968 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:45.400976 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:45.401216 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:45.401745 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:45.401750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:45.401753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:45.402489 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:45.402750 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:45.404176 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0620 03:05:45.404655 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":388,"threads_started":1}
I20260620 03:05:45.407418 2884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d204e6da3f941059733a229397f435d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38813 } }
I20260620 03:05:45.407912 2884 leader_election.cc:302] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d204e6da3f941059733a229397f435d; no voters:
I20260620 03:05:45.408524 2884 leader_election.cc:288] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:05:45.408674 2889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:05:45.409896 2889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d [term 1 LEADER]: Becoming Leader. State: Replica: 5d204e6da3f941059733a229397f435d, State: Running, Role: LEADER
I20260620 03:05:45.410368 2889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d204e6da3f941059733a229397f435d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38813 } }
I20260620 03:05:45.410645 2884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:05:45.413265 2890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d204e6da3f941059733a229397f435d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d204e6da3f941059733a229397f435d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38813 } } }
I20260620 03:05:45.413290 2891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d204e6da3f941059733a229397f435d. Latest consensus state: current_term: 1 leader_uuid: "5d204e6da3f941059733a229397f435d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d204e6da3f941059733a229397f435d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38813 } } }
I20260620 03:05:45.413760 2891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d [sys.catalog]: This master's current role is: LEADER
I20260620 03:05:45.413760 2890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d [sys.catalog]: This master's current role is: LEADER
I20260620 03:05:45.414837 2898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:05:45.421335 2898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:05:45.431421 2898 catalog_manager.cc:1347] Generated new cluster ID: 45d2cc97f0ef42dca128e1e8f38dc947
I20260620 03:05:45.431552 2898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:05:45.444722 2898 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:05:45.446475 2898 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:05:45.462648 2898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d204e6da3f941059733a229397f435d: Generated new TSK 0
I20260620 03:05:45.463672 2898 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:05:45.511196 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38813
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:45.830331 2910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:45.830678 2910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:45.830852 2910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:45.837994 2910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:45.838145 2910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:45.838331 2910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:05:45.847875 2910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:45.849196 2910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:45.850510 2910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:45.863744 2916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:45.863827 2915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:45.863943 2918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:45.864424 2910 server_base.cc:1034] running on GCE node
I20260620 03:05:45.865064 2910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:45.866077 2910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:45.867360 2910 hybrid_clock.cc:648] HybridClock initialized: now 1781924745867274 us; error 49 us; skew 500 ppm
I20260620 03:05:45.867692 2910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:45 dist-test-slave-xv68 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:45.871165 2910 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:05:45.872730 2910 webserver.cc:460] Webserver started at http://127.0.97.1:35893/ using document root <none> and password file <none>
I20260620 03:05:45.873423 2910 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:45.873572 2910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:45.873881 2910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:45.876814 2910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "f2cab8dc50f54abc8630fe550d8a4314"
format_stamp: "Formatted at 2026-06-20 03:05:45 on dist-test-slave-xv68"
server_key: "c13d89300fc30732628475740d7e95a1"
server_key_iv: "82efaf18b302ef66f60857f346aa445c"
server_key_version: "encryptionkey@0"
I20260620 03:05:45.877615 2910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f2cab8dc50f54abc8630fe550d8a4314"
format_stamp: "Formatted at 2026-06-20 03:05:45 on dist-test-slave-xv68"
server_key: "c13d89300fc30732628475740d7e95a1"
server_key_iv: "82efaf18b302ef66f60857f346aa445c"
server_key_version: "encryptionkey@0"
I20260620 03:05:45.882938 2910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 03:05:45.886565 2925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:45.888185 2910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 03:05:45.888368 2910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "f2cab8dc50f54abc8630fe550d8a4314"
format_stamp: "Formatted at 2026-06-20 03:05:45 on dist-test-slave-xv68"
server_key: "c13d89300fc30732628475740d7e95a1"
server_key_iv: "82efaf18b302ef66f60857f346aa445c"
server_key_version: "encryptionkey@0"
I20260620 03:05:45.888592 2910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:45.902182 2910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:45.906287 2910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:45.906667 2910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:45.907681 2910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:05:45.908991 2910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:45.909094 2910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:45.909237 2910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:45.909314 2910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:45.943230 2910 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39985
I20260620 03:05:45.943277 3037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39985 every 8 connection(s)
I20260620 03:05:45.944581 2910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:05:45.950438 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 2910
I20260620 03:05:45.950665 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:05:45.951568 388 external_mini_cluster.cc:1428] Setting key eb17a31a25e92d1848ae5f5e2754bf8b
Jun 20 03:05:45 dist-test-slave-xv68 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:45.964982 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:45.947166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44723 (local address 127.0.97.62:38813)
0620 03:05:45.947644 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:45.947649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:45.948688 (+ 1039us) server_negotiation.cc:407] Connection header received
0620 03:05:45.950975 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:45.950983 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:45.951069 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:45.951165 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:45.953006 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:45.953730 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:45.955328 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:45.955756 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:45.955899 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:45.959215 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:45.959220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:45.959223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:45.959252 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:45.961839 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:45.962700 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:45.962706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:45.962710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:45.962828 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:45.963269 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:45.963277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:45.963280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:45.963575 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:45.963732 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:45.964413 (+ 681us) server_negotiation.cc:299] Negotiation successful
0620 03:05:45.964617 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":271,"threads_started":1}
I20260620 03:05:45.966019 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:45.947793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38813 (local address 127.0.97.1:44723)
0620 03:05:45.948518 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:45.948556 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:45.950496 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:45.951496 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:45.951512 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:45.952258 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:45.952837 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:45.952854 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:45.953928 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:45.953932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:45.955147 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:45.955158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:45.955984 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:45.955990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:45.956222 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:45.956901 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:45.956923 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:45.958995 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:45.962000 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:45.962009 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:45.962027 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:45.962580 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:45.962974 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:45.962980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:45.962983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:45.963158 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:45.963701 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:45.963709 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:45.964051 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 03:05:45.964309 (+ 258us) client_negotiation.cc:240] Negotiation successful
0620 03:05:45.964812 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":276,"threads_started":1}
I20260620 03:05:45.967763 3038 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38813
I20260620 03:05:45.968111 3038 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 03:05:45.968904 3038 heartbeater.cc:511] Master 127.0.97.62:38813 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 03:05:45.971088 2849 ts_manager.cc:194] Registered new tserver with Master: f2cab8dc50f54abc8630fe550d8a4314 (127.0.97.1:39985)
I20260620 03:05:45.972536 2849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:05:45.992988 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38813
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:46.307835 3044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:46.308107 3044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:46.308198 3044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:46.315537 3044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:46.315640 3044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:46.315809 3044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:05:46.325781 3044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:46.327078 3044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:46.328411 3044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:46.341126 3049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:46.341200 3052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:46.341219 3050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:46.342418 3044 server_base.cc:1034] running on GCE node
I20260620 03:05:46.342970 3044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:46.343933 3044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:46.345156 3044 hybrid_clock.cc:648] HybridClock initialized: now 1781924746345105 us; error 61 us; skew 500 ppm
I20260620 03:05:46.345459 3044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:46 dist-test-slave-xv68 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:46.348563 3044 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:05:46.349977 3044 webserver.cc:460] Webserver started at http://127.0.97.2:36835/ using document root <none> and password file <none>
I20260620 03:05:46.350629 3044 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:46.350766 3044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:46.351055 3044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:46.354486 3044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "d3576964bfb448ab8ff07fe0c2b58345"
format_stamp: "Formatted at 2026-06-20 03:05:46 on dist-test-slave-xv68"
server_key: "41bcad808d24fa83be2f9f9fe24f56f4"
server_key_iv: "91644e60d6501c520d94ef8a6b7886ec"
server_key_version: "encryptionkey@0"
I20260620 03:05:46.355244 3044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d3576964bfb448ab8ff07fe0c2b58345"
format_stamp: "Formatted at 2026-06-20 03:05:46 on dist-test-slave-xv68"
server_key: "41bcad808d24fa83be2f9f9fe24f56f4"
server_key_iv: "91644e60d6501c520d94ef8a6b7886ec"
server_key_version: "encryptionkey@0"
I20260620 03:05:46.360184 3044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 03:05:46.363468 3059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:46.364801 3044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:05:46.364985 3044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "d3576964bfb448ab8ff07fe0c2b58345"
format_stamp: "Formatted at 2026-06-20 03:05:46 on dist-test-slave-xv68"
server_key: "41bcad808d24fa83be2f9f9fe24f56f4"
server_key_iv: "91644e60d6501c520d94ef8a6b7886ec"
server_key_version: "encryptionkey@0"
I20260620 03:05:46.365190 3044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:46.396327 3044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:46.400902 3044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:46.401201 3044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:46.402215 3044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:05:46.403666 3044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:46.403770 3044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:46.403925 3044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:46.403999 3044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:46.438395 3044 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40927
I20260620 03:05:46.438443 3171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40927 every 8 connection(s)
I20260620 03:05:46.439682 3044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:05:46.441071 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 3044
I20260620 03:05:46.441269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:05:46.441705 388 external_mini_cluster.cc:1428] Setting key 6b9687aaa70ed0a99405b5b5c8657cde
Jun 20 03:05:46 dist-test-slave-xv68 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 03:05:46.460521 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:46.442137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55759 (local address 127.0.97.62:38813)
0620 03:05:46.442345 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:46.442353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:46.443694 (+ 1341us) server_negotiation.cc:407] Connection header received
0620 03:05:46.446540 (+ 2846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:46.446544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:46.446604 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:46.446688 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:46.448747 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:46.449467 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:46.450980 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:46.451435 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:46.451572 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:46.454941 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:46.454946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:46.454949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:46.454980 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:46.457571 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:46.458247 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:46.458254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:46.458257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:46.458378 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:46.458815 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:46.458820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:46.458823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:46.459067 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:46.459212 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:46.459958 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 03:05:46.460180 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 03:05:46.461481 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:46.442702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38813 (local address 127.0.97.2:55759)
0620 03:05:46.443540 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:46.443586 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:46.446234 (+ 2648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:46.446952 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:46.446967 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:46.447761 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:46.448558 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:46.448584 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:46.449631 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:46.449637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:46.450834 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:46.450850 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:46.451654 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:46.451661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:46.451977 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:46.452695 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:46.452716 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:46.454733 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:46.457707 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:46.457716 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:46.457732 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:46.458130 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:46.458515 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:46.458520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:46.458524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:46.458697 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:46.459185 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:46.459194 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:46.459556 (+ 362us) client_negotiation.cc:769] Sending connection context
0620 03:05:46.459859 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 03:05:46.460363 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":294,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 03:05:46.462994 3172 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38813
I20260620 03:05:46.463337 3172 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:46.464170 3172 heartbeater.cc:511] Master 127.0.97.62:38813 requested a full tablet report, sending...
I20260620 03:05:46.465682 2849 ts_manager.cc:194] Registered new tserver with Master: d3576964bfb448ab8ff07fe0c2b58345 (127.0.97.2:40927)
I20260620 03:05:46.466434 2849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:05:46.486171 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38813
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:46.806066 3177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:46.806350 3177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:46.806442 3177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:46.813656 3177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:46.813750 3177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:46.813915 3177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:05:46.823520 3177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:46.824738 3177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:46.826045 3177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:46.839077 3185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:46.839056 3182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:46.839115 3183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:46.840478 3177 server_base.cc:1034] running on GCE node
I20260620 03:05:46.841095 3177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:46.841997 3177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:46.843214 3177 hybrid_clock.cc:648] HybridClock initialized: now 1781924746843141 us; error 59 us; skew 500 ppm
I20260620 03:05:46.843568 3177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:46 dist-test-slave-xv68 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:46.846678 3177 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:05:46.848140 3177 webserver.cc:460] Webserver started at http://127.0.97.3:39547/ using document root <none> and password file <none>
I20260620 03:05:46.848814 3177 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:46.848960 3177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:46.849279 3177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:46.852178 3177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "4724a4557dd94609870a2e422a4d29ce"
format_stamp: "Formatted at 2026-06-20 03:05:46 on dist-test-slave-xv68"
server_key: "80cc4cad8cf23d75c19f90d4c61c1ff4"
server_key_iv: "287582b0d8575407b86ae977b4c738e2"
server_key_version: "encryptionkey@0"
I20260620 03:05:46.853027 3177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4724a4557dd94609870a2e422a4d29ce"
format_stamp: "Formatted at 2026-06-20 03:05:46 on dist-test-slave-xv68"
server_key: "80cc4cad8cf23d75c19f90d4c61c1ff4"
server_key_iv: "287582b0d8575407b86ae977b4c738e2"
server_key_version: "encryptionkey@0"
I20260620 03:05:46.858160 3177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:05:46.861657 3192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:46.862962 3177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 03:05:46.863127 3177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "4724a4557dd94609870a2e422a4d29ce"
format_stamp: "Formatted at 2026-06-20 03:05:46 on dist-test-slave-xv68"
server_key: "80cc4cad8cf23d75c19f90d4c61c1ff4"
server_key_iv: "287582b0d8575407b86ae977b4c738e2"
server_key_version: "encryptionkey@0"
I20260620 03:05:46.863371 3177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:46.885665 3177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:46.890031 3177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:46.890323 3177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:46.891252 3177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:05:46.892649 3177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:46.892761 3177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:46.892915 3177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:46.892985 3177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:46.926537 3177 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39379
I20260620 03:05:46.926626 3304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39379 every 8 connection(s)
I20260620 03:05:46.927937 3177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:05:46.934078 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 3177
I20260620 03:05:46.934278 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:05:46.934832 388 external_mini_cluster.cc:1428] Setting key aae66687a6d8175febb5bafeec3635de
Jun 20 03:05:46 dist-test-slave-xv68 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:46.949609 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:46.930737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53933 (local address 127.0.97.62:38813)
0620 03:05:46.930933 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:46.930938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:46.932360 (+ 1422us) server_negotiation.cc:407] Connection header received
0620 03:05:46.934506 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:46.934510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:46.934580 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:46.934673 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:46.936659 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:46.937511 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:46.939058 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:46.939530 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:46.939635 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:46.943016 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:46.943021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:46.943026 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:46.943075 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:46.946197 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:46.946904 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:46.946910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:46.946914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:46.947051 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:46.947593 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:46.947598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:46.947600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:46.947940 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:46.948093 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:46.949046 (+ 953us) server_negotiation.cc:299] Negotiation successful
0620 03:05:46.949283 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 03:05:46.950636 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:46.931427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38813 (local address 127.0.97.3:53933)
0620 03:05:46.932191 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:46.932231 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:46.934168 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:46.934984 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:46.934996 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:46.935656 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:46.936509 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:46.936531 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:46.937694 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:46.937698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:46.938912 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:46.938924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:46.939744 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:46.939750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:46.940026 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:46.940676 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:46.940699 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:46.942812 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:46.946338 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:46.946345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:46.946356 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:46.946789 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:46.947182 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:46.947186 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:46.947189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:46.947478 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:46.948105 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:46.948114 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:46.948524 (+ 410us) client_negotiation.cc:769] Sending connection context
0620 03:05:46.948844 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 03:05:46.949357 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":284,"threads_started":1}
I20260620 03:05:46.951993 3305 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38813
I20260620 03:05:46.952309 3305 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:46.953079 3305 heartbeater.cc:511] Master 127.0.97.62:38813 requested a full tablet report, sending...
I20260620 03:05:46.954535 2849 ts_manager.cc:194] Registered new tserver with Master: 4724a4557dd94609870a2e422a4d29ce (127.0.97.3:39379)
I20260620 03:05:46.955353 2849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53933
I20260620 03:05:46.962706 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:05:46.975572 3038 heartbeater.cc:503] Master 127.0.97.62:38813 was elected leader, sending a full tablet report...
W20260620 03:05:46.993520 3040 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:05:46.990736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58476 (local address 127.0.97.62:38813)
0620 03:05:46.990979 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:46.990984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:46.991462 (+ 478us) server_negotiation.cc:407] Connection header received
0620 03:05:46.991611 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:46.991615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:46.991691 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:46.991785 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:46.993224 (+ 1439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58476: BlockingRecv error: recv got EOF from 127.0.0.1:58476 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 03:05:46.997476 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 2910
W20260620 03:05:47.000447 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 2910: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:39985: server requires authentication, but client does not have Kerberos credentials available
I20260620 03:05:47.000636 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 2910
I20260620 03:05:47.017939 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 3044
W20260620 03:05:47.020792 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 3044: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:40927: server requires authentication, but client does not have Kerberos credentials available
I20260620 03:05:47.021027 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 3044
I20260620 03:05:47.037039 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 3177
W20260620 03:05:47.040123 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 3177: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:39379: server requires authentication, but client does not have Kerberos credentials available
I20260620 03:05:47.040277 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 3177
I20260620 03:05:47.055066 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 2817
I20260620 03:05:47.056336 2879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:05:47.157459 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 2817
2026-06-20T03:05:47Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3495 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:05:47 dist-test-slave-xv68 krb5kdc[3322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:05:47 dist-test-slave-xv68 krb5kdc[3322](info): set up 2 sockets
Jun 20 03:05:47 dist-test-slave-xv68 krb5kdc[3322](info): commencing operation
krb5kdc: starting...
W20260620 03:05:48.263666 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:05:48 dist-test-slave-xv68 krb5kdc[3322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:05:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:05:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:05:48.455096 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41259
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:48.767041 3335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:48.767400 3335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:48.767563 3335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:48.774578 3335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:05:48.774731 3335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:48.774838 3335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:48.774950 3335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:05:48.775041 3335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:05:48.784634 3335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:48.785945 3335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:48.787277 3335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:48.800045 3340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:48.800004 3343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:48.799965 3341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:48.801231 3335 server_base.cc:1034] running on GCE node
I20260620 03:05:48.802080 3335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:48.803521 3335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:48.804746 3335 hybrid_clock.cc:648] HybridClock initialized: now 1781924748804694 us; error 44 us; skew 500 ppm
I20260620 03:05:48.805104 3335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:48 dist-test-slave-xv68 krb5kdc[3322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:48.811285 3335 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:05:48.812918 3335 webserver.cc:460] Webserver started at http://127.0.97.62:40767/ using document root <none> and password file <none>
I20260620 03:05:48.813678 3335 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:48.813828 3335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:48.814136 3335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:48.817219 3335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "68063dd4d1b049178209bff618a003fa"
format_stamp: "Formatted at 2026-06-20 03:05:48 on dist-test-slave-xv68"
server_key: "ec4688241ebb3a959b94a16ac9d5dcee"
server_key_iv: "6641e46ddaae3b5b8a5f770348bd9f7a"
server_key_version: "encryptionkey@0"
I20260620 03:05:48.818037 3335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "68063dd4d1b049178209bff618a003fa"
format_stamp: "Formatted at 2026-06-20 03:05:48 on dist-test-slave-xv68"
server_key: "ec4688241ebb3a959b94a16ac9d5dcee"
server_key_iv: "6641e46ddaae3b5b8a5f770348bd9f7a"
server_key_version: "encryptionkey@0"
I20260620 03:05:48.823403 3335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 03:05:48.826781 3350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:48.828197 3335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 03:05:48.828384 3335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "68063dd4d1b049178209bff618a003fa"
format_stamp: "Formatted at 2026-06-20 03:05:48 on dist-test-slave-xv68"
server_key: "ec4688241ebb3a959b94a16ac9d5dcee"
server_key_iv: "6641e46ddaae3b5b8a5f770348bd9f7a"
server_key_version: "encryptionkey@0"
I20260620 03:05:48.828598 3335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:48.865644 3335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:48.869599 3335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:48.869889 3335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:48.888015 3335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41259
I20260620 03:05:48.888055 3401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41259 every 8 connection(s)
I20260620 03:05:48.889379 3335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:05:48.893045 3402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:48.893402 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 3335
I20260620 03:05:48.893617 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:05:48.894250 388 external_mini_cluster.cc:1428] Setting key c66ca20e349110bfb1be8b40e3fff6c4
I20260620 03:05:48.904834 3402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa: Bootstrap starting.
Jun 20 03:05:48 dist-test-slave-xv68 krb5kdc[3322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:48.908087 3402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:48.909303 3402 log.cc:824] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:48.912261 3402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa: No bootstrap required, opened a new log
I20260620 03:05:48.916056 3405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:48.896164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.0.97.62:41259)
0620 03:05:48.897134 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:48.897146 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:48.897193 (+ 47us) server_negotiation.cc:407] Connection header received
0620 03:05:48.898990 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:48.899012 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:48.899471 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:48.899863 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:48.901279 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:48.902572 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:48.903879 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:48.904560 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:48.904759 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:48.908937 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:48.908944 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:48.908980 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:48.909013 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:48.912278 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:48.912923 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:48.912929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:48.912934 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:48.913087 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:48.913466 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:48.913470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:48.913472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:48.913944 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:48.914257 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:48.914673 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 03:05:48.915052 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":470,"threads_started":1}
I20260620 03:05:48.919580 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:05:48.920185 3402 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68063dd4d1b049178209bff618a003fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41259 } }
I20260620 03:05:48.920517 3402 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:48.920621 3402 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68063dd4d1b049178209bff618a003fa, State: Initialized, Role: FOLLOWER
I20260620 03:05:48.921134 3402 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68063dd4d1b049178209bff618a003fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41259 } }
I20260620 03:05:48.921350 3402 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:05:48.921483 3402 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:05:48.921651 3402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:48.923761 3402 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68063dd4d1b049178209bff618a003fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41259 } }
I20260620 03:05:48.924222 3402 leader_election.cc:302] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68063dd4d1b049178209bff618a003fa; no voters:
I20260620 03:05:48.924841 3402 leader_election.cc:288] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:05:48.924947 3407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:05:48.927162 3402 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:05:48.927412 3407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa [term 1 LEADER]: Becoming Leader. State: Replica: 68063dd4d1b049178209bff618a003fa, State: Running, Role: LEADER
I20260620 03:05:48.927874 3407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68063dd4d1b049178209bff618a003fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41259 } }
I20260620 03:05:48.930545 3408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68063dd4d1b049178209bff618a003fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68063dd4d1b049178209bff618a003fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41259 } } }
I20260620 03:05:48.930675 3410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68063dd4d1b049178209bff618a003fa. Latest consensus state: current_term: 1 leader_uuid: "68063dd4d1b049178209bff618a003fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68063dd4d1b049178209bff618a003fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41259 } } }
I20260620 03:05:48.930929 3408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa [sys.catalog]: This master's current role is: LEADER
I20260620 03:05:48.930934 3410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa [sys.catalog]: This master's current role is: LEADER
I20260620 03:05:48.945000 3416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:05:48.950659 3416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:05:48.959259 3426 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 03:05:48.960599 3416 catalog_manager.cc:1347] Generated new cluster ID: d2b51657f7924ffa937e81215aa41527
I20260620 03:05:48.960698 3416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:05:48.970312 3405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:48.963595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48690 (local address 127.0.97.62:41259)
0620 03:05:48.963758 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:48.963769 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:48.963824 (+ 55us) server_negotiation.cc:407] Connection header received
0620 03:05:48.964079 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:48.964089 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:48.964148 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:48.964259 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:48.965204 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:48.965937 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:48.967221 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:48.967710 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:48.967848 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:48.968934 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:48.968939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:48.968942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:05:48.969524 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:48.969579 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:05:48.969609 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:48.969735 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:48.969863 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 03:05:48.969949 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 03:05:48.992483 3416 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:05:48.994356 3416 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:05:49.003857 3416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68063dd4d1b049178209bff618a003fa: Generated new TSK 0
I20260620 03:05:49.004911 3416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:05:49.015447 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 3335
I20260620 03:05:49.016624 3397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:05:49.107954 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 3335
2026-06-20T03:05:49Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1944 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:05:49 dist-test-slave-xv68 krb5kdc[3436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:05:49 dist-test-slave-xv68 krb5kdc[3436](info): set up 2 sockets
Jun 20 03:05:49 dist-test-slave-xv68 krb5kdc[3436](info): commencing operation
krb5kdc: starting...
W20260620 03:05:50.183691 388 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:05:50 dist-test-slave-xv68 krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:05:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:05:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:05:50.370671 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46749
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:50.683761 3449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:50.684060 3449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:50.684158 3449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:50.691421 3449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:05:50.691524 3449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:50.691569 3449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:50.691627 3449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:05:50.691670 3449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:05:50.701260 3449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:50.702541 3449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:50.703836 3449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:50.715757 3454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:50.715783 3455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:50.715808 3457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:50.716531 3449 server_base.cc:1034] running on GCE node
I20260620 03:05:50.717253 3449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:50.718575 3449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:50.719792 3449 hybrid_clock.cc:648] HybridClock initialized: now 1781924750719763 us; error 53 us; skew 500 ppm
I20260620 03:05:50.720140 3449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:50 dist-test-slave-xv68 krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:50.723083 3449 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:05:50.724450 3449 webserver.cc:460] Webserver started at http://127.0.97.62:39159/ using document root <none> and password file <none>
I20260620 03:05:50.725051 3449 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:50.725147 3449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:50.725379 3449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:50.728227 3449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "adb45289df7a432c80692333a48c0182"
format_stamp: "Formatted at 2026-06-20 03:05:50 on dist-test-slave-xv68"
server_key: "f227eb3a05e3b1da163b0586b2840efa"
server_key_iv: "a2bd6f1f79aac520ecd02f759872c4e0"
server_key_version: "encryptionkey@0"
I20260620 03:05:50.728904 3449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "adb45289df7a432c80692333a48c0182"
format_stamp: "Formatted at 2026-06-20 03:05:50 on dist-test-slave-xv68"
server_key: "f227eb3a05e3b1da163b0586b2840efa"
server_key_iv: "a2bd6f1f79aac520ecd02f759872c4e0"
server_key_version: "encryptionkey@0"
I20260620 03:05:50.734083 3449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:05:50.737219 3464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:50.738546 3449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:05:50.738715 3449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "adb45289df7a432c80692333a48c0182"
format_stamp: "Formatted at 2026-06-20 03:05:50 on dist-test-slave-xv68"
server_key: "f227eb3a05e3b1da163b0586b2840efa"
server_key_iv: "a2bd6f1f79aac520ecd02f759872c4e0"
server_key_version: "encryptionkey@0"
I20260620 03:05:50.738888 3449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:50.758756 3449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:50.762775 3449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:50.763049 3449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:50.781201 3449 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46749
I20260620 03:05:50.781286 3515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46749 every 8 connection(s)
I20260620 03:05:50.782546 3449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:05:50.786180 3516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:50.788174 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 3449
I20260620 03:05:50.788379 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:05:50.789179 388 external_mini_cluster.cc:1428] Setting key d80dc1102fc99bf03c112fac98ae24d0
I20260620 03:05:50.798719 3516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182: Bootstrap starting.
I20260620 03:05:50.802826 3516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182: Neither blocks nor log segments found. Creating new log.
Jun 20 03:05:50 dist-test-slave-xv68 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:50.804143 3516 log.cc:824] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:50.807509 3516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182: No bootstrap required, opened a new log
I20260620 03:05:50.812531 3519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:50.791228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.0.97.62:46749)
0620 03:05:50.792106 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:50.792118 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:50.792158 (+ 40us) server_negotiation.cc:407] Connection header received
0620 03:05:50.793820 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:50.793850 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:50.794230 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:50.794547 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:50.796261 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:50.797622 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:50.799411 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:50.800097 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:50.800238 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:50.805223 (+ 4985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:50.805236 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:50.805264 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:50.805322 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:50.808532 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:50.809330 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:50.809336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:50.809341 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:50.809518 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:50.809920 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:50.809923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:50.809925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:50.810408 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:50.810651 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:50.811074 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 03:05:50.811510 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":385,"threads_started":1}
I20260620 03:05:50.815924 3516 raft_consensus.cc:348] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb45289df7a432c80692333a48c0182" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46749 } }
I20260620 03:05:50.816267 3516 raft_consensus.cc:374] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:50.816370 3516 raft_consensus.cc:729] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb45289df7a432c80692333a48c0182, State: Initialized, Role: FOLLOWER
I20260620 03:05:50.816900 3516 consensus_queue.cc:260] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb45289df7a432c80692333a48c0182" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46749 } }
I20260620 03:05:50.817135 3516 raft_consensus.cc:388] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:05:50.817286 3516 raft_consensus.cc:482] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:05:50.817461 3516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:50.819550 3516 raft_consensus.cc:504] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb45289df7a432c80692333a48c0182" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46749 } }
I20260620 03:05:50.819994 3516 leader_election.cc:302] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adb45289df7a432c80692333a48c0182; no voters:
I20260620 03:05:50.820530 3516 leader_election.cc:288] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:05:50.820714 3521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:05:50.821521 3521 raft_consensus.cc:686] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182 [term 1 LEADER]: Becoming Leader. State: Replica: adb45289df7a432c80692333a48c0182, State: Running, Role: LEADER
I20260620 03:05:50.822072 3521 consensus_queue.cc:237] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb45289df7a432c80692333a48c0182" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46749 } }
I20260620 03:05:50.823391 3516 sys_catalog.cc:564] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:05:50.824796 3523 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adb45289df7a432c80692333a48c0182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb45289df7a432c80692333a48c0182" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46749 } } }
I20260620 03:05:50.824786 3522 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adb45289df7a432c80692333a48c0182. Latest consensus state: current_term: 1 leader_uuid: "adb45289df7a432c80692333a48c0182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb45289df7a432c80692333a48c0182" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46749 } } }
I20260620 03:05:50.825215 3523 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182 [sys.catalog]: This master's current role is: LEADER
I20260620 03:05:50.825268 3522 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182 [sys.catalog]: This master's current role is: LEADER
I20260620 03:05:50.828758 3526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:05:50.835242 3526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:05:50.846640 3526 catalog_manager.cc:1347] Generated new cluster ID: 21a73823a82648ce9004410113ceb871
I20260620 03:05:50.846827 3526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:05:50.857856 3526 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:05:50.859087 3526 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:05:50.867018 3526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adb45289df7a432c80692333a48c0182: Generated new TSK 0
I20260620 03:05:50.867854 3526 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:05:50.913502 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46749
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:51.236840 3542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:51.237112 3542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:51.237258 3542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:51.244892 3542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:51.244990 3542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:51.245157 3542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:05:51.254694 3542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:51.255985 3542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:51.257288 3542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:51.270321 3548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:51.270359 3550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:51.270385 3547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:51.271397 3542 server_base.cc:1034] running on GCE node
I20260620 03:05:51.272033 3542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:51.273015 3542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:51.274236 3542 hybrid_clock.cc:648] HybridClock initialized: now 1781924751274176 us; error 62 us; skew 500 ppm
I20260620 03:05:51.274540 3542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:51 dist-test-slave-xv68 krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:51.278019 3542 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:05:51.279596 3542 webserver.cc:460] Webserver started at http://127.0.97.1:43243/ using document root <none> and password file <none>
I20260620 03:05:51.280288 3542 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:51.280431 3542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:51.280731 3542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:51.283761 3542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "7a17870095ff4cecb85fdae3a3cd0688"
format_stamp: "Formatted at 2026-06-20 03:05:51 on dist-test-slave-xv68"
server_key: "b045ffbedf2948c99744bdd2655fc492"
server_key_iv: "2c24bd574712cc9400714254aff3d8fa"
server_key_version: "encryptionkey@0"
I20260620 03:05:51.284548 3542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7a17870095ff4cecb85fdae3a3cd0688"
format_stamp: "Formatted at 2026-06-20 03:05:51 on dist-test-slave-xv68"
server_key: "b045ffbedf2948c99744bdd2655fc492"
server_key_iv: "2c24bd574712cc9400714254aff3d8fa"
server_key_version: "encryptionkey@0"
I20260620 03:05:51.289494 3542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 03:05:51.292807 3557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:51.294094 3542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 03:05:51.294261 3542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "7a17870095ff4cecb85fdae3a3cd0688"
format_stamp: "Formatted at 2026-06-20 03:05:51 on dist-test-slave-xv68"
server_key: "b045ffbedf2948c99744bdd2655fc492"
server_key_iv: "2c24bd574712cc9400714254aff3d8fa"
server_key_version: "encryptionkey@0"
I20260620 03:05:51.294450 3542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:51.337733 3542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:51.342231 3542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:51.342558 3542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:51.343641 3542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:05:51.344898 3542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:51.345000 3542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:51.345144 3542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:51.345218 3542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:51.377458 3542 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39859
I20260620 03:05:51.377547 3669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39859 every 8 connection(s)
I20260620 03:05:51.378679 3542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:05:51.383744 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 3542
I20260620 03:05:51.383949 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:05:51.384603 388 external_mini_cluster.cc:1428] Setting key 9a6fd594f50362e3bd6e97f84f75eeb8
Jun 20 03:05:51 dist-test-slave-xv68 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:51.402379 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:51.381324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41497 (local address 127.0.97.62:46749)
0620 03:05:51.381774 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:51.381780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:51.382785 (+ 1005us) server_negotiation.cc:407] Connection header received
0620 03:05:51.384729 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:51.384735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:51.384836 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:51.384961 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:51.386649 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:51.387816 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:51.389358 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:51.390089 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:51.390236 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:51.394322 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:51.394329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:51.394334 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:51.394399 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:51.398704 (+ 4305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:51.399786 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:51.399792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:51.399795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:51.399972 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:51.400501 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:51.400507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:51.400510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:51.400850 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:51.400981 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:51.401802 (+ 821us) server_negotiation.cc:299] Negotiation successful
0620 03:05:51.402012 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":226,"threads_started":1}
I20260620 03:05:51.403560 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:51.381889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46749 (local address 127.0.97.1:41497)
0620 03:05:51.382624 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:51.382661 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:51.384454 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:51.385154 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:51.385169 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:51.385812 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:51.386488 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:51.386507 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:51.387969 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:51.387973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:51.389205 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:51.389216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:51.390298 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:51.390302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:51.390488 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:51.391174 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:51.391196 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:51.393985 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:51.398883 (+ 4898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:51.398895 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:51.398913 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:51.399595 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:51.400089 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:51.400095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:51.400098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:51.400366 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:51.401089 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:51.401099 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:51.401432 (+ 333us) client_negotiation.cc:769] Sending connection context
0620 03:05:51.401699 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 03:05:51.402230 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":307,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 03:05:51.405340 3670 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46749
I20260620 03:05:51.405663 3670 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:51.406461 3670 heartbeater.cc:511] Master 127.0.97.62:46749 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 03:05:51.408793 3481 ts_manager.cc:194] Registered new tserver with Master: 7a17870095ff4cecb85fdae3a3cd0688 (127.0.97.1:39859)
I20260620 03:05:51.410914 3481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:05:51.430915 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46749
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:51.747246 3676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:51.747597 3676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:51.747771 3676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:51.754904 3676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:51.755074 3676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:51.755268 3676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:05:51.765228 3676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:51.766861 3676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:51.768216 3676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:51.781620 3682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:51.781940 3684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:51.782302 3676 server_base.cc:1034] running on GCE node
W20260620 03:05:51.787586 3681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:51.788381 3676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:51.789388 3676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:51.790624 3676 hybrid_clock.cc:648] HybridClock initialized: now 1781924751790554 us; error 63 us; skew 500 ppm
I20260620 03:05:51.790954 3676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:51 dist-test-slave-xv68 krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:51.794140 3676 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:05:51.795725 3676 webserver.cc:460] Webserver started at http://127.0.97.2:45091/ using document root <none> and password file <none>
I20260620 03:05:51.796416 3676 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:51.796536 3676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:51.796849 3676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:51.800034 3676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "6c115d4790ea4579942ea9b855123fc0"
format_stamp: "Formatted at 2026-06-20 03:05:51 on dist-test-slave-xv68"
server_key: "2a2d7336b0c0d56d781b1090b47cc792"
server_key_iv: "a4ce66e8a4fb7f81e199c092abcac6a3"
server_key_version: "encryptionkey@0"
I20260620 03:05:51.800808 3676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6c115d4790ea4579942ea9b855123fc0"
format_stamp: "Formatted at 2026-06-20 03:05:51 on dist-test-slave-xv68"
server_key: "2a2d7336b0c0d56d781b1090b47cc792"
server_key_iv: "a4ce66e8a4fb7f81e199c092abcac6a3"
server_key_version: "encryptionkey@0"
I20260620 03:05:51.806504 3676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:05:51.810099 3691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:51.811482 3676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 03:05:51.811611 3676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "6c115d4790ea4579942ea9b855123fc0"
format_stamp: "Formatted at 2026-06-20 03:05:51 on dist-test-slave-xv68"
server_key: "2a2d7336b0c0d56d781b1090b47cc792"
server_key_iv: "a4ce66e8a4fb7f81e199c092abcac6a3"
server_key_version: "encryptionkey@0"
I20260620 03:05:51.811782 3676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:51.825197 3676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:51.829371 3676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:51.829614 3676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:51.830480 3676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:05:51.831794 3676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:51.831866 3676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:51.831961 3676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:51.832010 3676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:51.865339 3676 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46001
I20260620 03:05:51.865366 3803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46001 every 8 connection(s)
I20260620 03:05:51.866631 3676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:05:51.868501 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 3676
I20260620 03:05:51.868667 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:05:51.869256 388 external_mini_cluster.cc:1428] Setting key 0007591c9aeaff4752313aba9e56edb8
Jun 20 03:05:51 dist-test-slave-xv68 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 03:05:51.888563 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:51.869374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54607 (local address 127.0.97.62:46749)
0620 03:05:51.869556 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:51.869561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:51.871206 (+ 1645us) server_negotiation.cc:407] Connection header received
0620 03:05:51.873035 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:51.873043 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:51.873132 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:51.873259 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:51.874741 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:51.875920 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:51.878530 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:51.878952 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:51.879103 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:51.882817 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:51.882822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:51.882825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:51.882856 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:51.885527 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:51.886248 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:51.886255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:51.886258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:51.886391 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:51.886827 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:51.886833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:51.886837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:51.887088 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:51.887245 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:51.887996 (+ 751us) server_negotiation.cc:299] Negotiation successful
0620 03:05:51.888225 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 03:05:51.889638 3806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:51.869971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46749 (local address 127.0.97.2:54607)
0620 03:05:51.871021 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:51.871071 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:51.872771 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:51.873458 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:51.873466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:51.873991 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:51.874568 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:51.874586 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:51.876107 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:51.876111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:51.878359 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:51.878376 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:51.879202 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:51.879212 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:51.879600 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:51.880386 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:51.880409 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:51.882562 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:51.885681 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:51.885690 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:51.885709 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:51.886123 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:51.886509 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:51.886514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:51.886519 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:51.886718 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:51.887220 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:51.887229 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:51.887603 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 03:05:51.887918 (+ 315us) client_negotiation.cc:240] Negotiation successful
0620 03:05:51.888454 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":457,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 03:05:51.891139 3804 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46749
I20260620 03:05:51.891494 3804 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:51.892248 3804 heartbeater.cc:511] Master 127.0.97.62:46749 requested a full tablet report, sending...
I20260620 03:05:51.893726 3481 ts_manager.cc:194] Registered new tserver with Master: 6c115d4790ea4579942ea9b855123fc0 (127.0.97.2:46001)
I20260620 03:05:51.894569 3481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:05:51.914081 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46749
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:52.233027 3809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:52.233306 3809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:52.233398 3809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:52.240782 3809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:52.240890 3809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:52.241014 3809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:05:52.250643 3809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:52.251909 3809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:52.253165 3809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:52.265978 3817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:52.266088 3815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:52.266099 3814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:52.271891 3809 server_base.cc:1034] running on GCE node
I20260620 03:05:52.272533 3809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:52.273444 3809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:52.274695 3809 hybrid_clock.cc:648] HybridClock initialized: now 1781924752274642 us; error 68 us; skew 500 ppm
I20260620 03:05:52.275065 3809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:52 dist-test-slave-xv68 krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:52.278343 3809 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:05:52.279968 3809 webserver.cc:460] Webserver started at http://127.0.97.3:46161/ using document root <none> and password file <none>
I20260620 03:05:52.280792 3809 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:52.280941 3809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:52.281260 3809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:52.284493 3809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "6d284b465075435ea45b655c3fff8289"
format_stamp: "Formatted at 2026-06-20 03:05:52 on dist-test-slave-xv68"
server_key: "8cb100aebe1ae47f81f7483981b0f70b"
server_key_iv: "069464e3a1b9955e071ed689860e009a"
server_key_version: "encryptionkey@0"
I20260620 03:05:52.285411 3809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6d284b465075435ea45b655c3fff8289"
format_stamp: "Formatted at 2026-06-20 03:05:52 on dist-test-slave-xv68"
server_key: "8cb100aebe1ae47f81f7483981b0f70b"
server_key_iv: "069464e3a1b9955e071ed689860e009a"
server_key_version: "encryptionkey@0"
I20260620 03:05:52.290755 3809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 03:05:52.294231 3824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:52.295588 3809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:05:52.295758 3809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "6d284b465075435ea45b655c3fff8289"
format_stamp: "Formatted at 2026-06-20 03:05:52 on dist-test-slave-xv68"
server_key: "8cb100aebe1ae47f81f7483981b0f70b"
server_key_iv: "069464e3a1b9955e071ed689860e009a"
server_key_version: "encryptionkey@0"
I20260620 03:05:52.295959 3809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:52.316864 3809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:52.320866 3809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:52.321168 3809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:52.322144 3809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:05:52.323627 3809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:52.323735 3809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:52.323882 3809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:52.323953 3809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:52.357618 3809 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34663
I20260620 03:05:52.357703 3936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34663 every 8 connection(s)
I20260620 03:05:52.358861 3809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:05:52.362151 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 3809
I20260620 03:05:52.362357 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:05:52.363018 388 external_mini_cluster.cc:1428] Setting key a69b2a849430ce55abdd6213ab9add21
Jun 20 03:05:52 dist-test-slave-xv68 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:52.380295 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:52.361557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44535 (local address 127.0.97.62:46749)
0620 03:05:52.361754 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:52.361758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:52.363095 (+ 1337us) server_negotiation.cc:407] Connection header received
0620 03:05:52.365529 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:52.365537 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:52.365629 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:52.365734 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:52.367537 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:52.368358 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:52.369862 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:52.370241 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:52.370332 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:52.373781 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:52.373786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:52.373789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:52.373822 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:52.376822 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:52.377784 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:52.377790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:52.377793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:52.377954 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:52.378454 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:52.378459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:52.378462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:52.378805 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:52.378939 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:52.379726 (+ 787us) server_negotiation.cc:299] Negotiation successful
0620 03:05:52.379938 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 03:05:52.381151 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:52.362056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46749 (local address 127.0.97.3:44535)
0620 03:05:52.362900 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:52.362960 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:52.365209 (+ 2249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:52.365981 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:52.365990 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:52.366755 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:52.367362 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:52.367385 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:52.368497 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:52.368501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:52.369732 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:52.369743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:52.370423 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:52.370428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:52.370631 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:52.371235 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:52.371266 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:52.373559 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:52.376985 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:52.376997 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:52.377017 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:52.377608 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:52.378103 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:52.378108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:52.378112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:52.378325 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:52.378954 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:52.378963 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:52.379286 (+ 323us) client_negotiation.cc:769] Sending connection context
0620 03:05:52.379576 (+ 290us) client_negotiation.cc:240] Negotiation successful
0620 03:05:52.379997 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":335,"threads_started":1}
I20260620 03:05:52.382516 3937 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46749
I20260620 03:05:52.382836 3937 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:52.383642 3937 heartbeater.cc:511] Master 127.0.97.62:46749 requested a full tablet report, sending...
I20260620 03:05:52.385015 3481 ts_manager.cc:194] Registered new tserver with Master: 6d284b465075435ea45b655c3fff8289 (127.0.97.3:34663)
I20260620 03:05:52.385803 3481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44535
I20260620 03:05:52.391278 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:05:52.413868 3670 heartbeater.cc:503] Master 127.0.97.62:46749 was elected leader, sending a full tablet report...
Jun 20 03:05:52 dist-test-slave-xv68 krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924752, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 03:05:52 dist-test-slave-xv68 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924752, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:52.446861 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:52.433577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.0.97.62:46749)
0620 03:05:52.433761 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:52.433766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:52.434219 (+ 453us) server_negotiation.cc:407] Connection header received
0620 03:05:52.434356 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:52.434360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:52.434423 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:52.434515 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:52.435916 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:52.436631 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:52.437919 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:52.438347 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:52.438446 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:52.442154 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:52.442159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:52.442162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:52.442192 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:52.444730 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:52.445315 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:52.445319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:52.445321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:52.445425 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:52.445776 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:52.445780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:52.445782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:52.446058 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:52.446231 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:52.446475 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 03:05:52.446629 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 03:05:52.447515 3481 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41668
I20260620 03:05:52.449290 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 3542
Jun 20 03:05:52 dist-test-slave-xv68 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924752, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:05:52.464560 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:52.450286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58270 (local address 127.0.97.1:39859)
0620 03:05:52.450932 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:52.450939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:52.450969 (+ 30us) server_negotiation.cc:407] Connection header received
0620 03:05:52.451019 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:52.451025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:52.451228 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:52.451435 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:52.452466 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:52.453417 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:52.454518 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:52.455102 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:52.455208 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:52.459087 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:52.459099 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:52.459118 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:52.459168 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:52.462100 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:52.462731 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:52.462739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:52.462741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:52.462864 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:52.463202 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:52.463206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:52.463208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:52.463613 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:52.463838 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:52.464096 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 03:05:52.464285 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":333,"threads_started":1}
W20260620 03:05:52.465821 3665 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58270
W20260620 03:05:52.466918 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 3542: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:05:52.467123 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 3542
I20260620 03:05:52.482602 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 3676
Jun 20 03:05:52 dist-test-slave-xv68 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924752, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:05:52.498673 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:52.483488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55042 (local address 127.0.97.2:46001)
0620 03:05:52.483981 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:52.483988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:52.484013 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:05:52.484076 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:52.484083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:52.484288 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:52.484454 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:52.485674 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:52.486655 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:52.488033 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:52.488579 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:52.488727 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:52.492842 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:52.492853 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:52.492868 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:52.492907 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:52.495762 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:52.496412 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:52.496420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:52.496422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:52.496535 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:52.496913 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:52.496922 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:52.496924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:52.497364 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:52.497715 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:52.498051 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 03:05:52.498323 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":237,"threads_started":1}
W20260620 03:05:52.500061 3799 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55042
W20260620 03:05:52.501259 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 3676: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:05:52.501408 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 3676
I20260620 03:05:52.516680 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 3809
Jun 20 03:05:52 dist-test-slave-xv68 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924752, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:05:52.532582 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:52.517584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50244 (local address 127.0.97.3:34663)
0620 03:05:52.518138 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:52.518145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:52.518174 (+ 29us) server_negotiation.cc:407] Connection header received
0620 03:05:52.518235 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:52.518241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:52.518445 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:52.518575 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:52.519905 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:52.520929 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:52.522131 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:52.522692 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:52.522796 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:52.527076 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:52.527087 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:52.527101 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:52.527135 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:52.529821 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:52.530449 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:52.530456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:52.530458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:52.530566 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:52.530924 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:52.530927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:52.530929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:52.531253 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:52.531593 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:52.531880 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 03:05:52.532085 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":246,"threads_started":1}
W20260620 03:05:52.533758 3932 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50244
W20260620 03:05:52.534787 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 3809: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:05:52.534955 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 3809
I20260620 03:05:52.550733 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 3449
I20260620 03:05:52.551934 3511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:05:52.653934 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 3449
2026-06-20T03:05:52Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3552 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:05:52 dist-test-slave-xv68 krb5kdc[3954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:05:52 dist-test-slave-xv68 krb5kdc[3954](info): set up 2 sockets
Jun 20 03:05:52 dist-test-slave-xv68 krb5kdc[3954](info): commencing operation
krb5kdc: starting...
W20260620 03:05:53.743718 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:05:53 dist-test-slave-xv68 krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:05:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:05:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:05:53.931547 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42251
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:54.243968 3967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:54.244308 3967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:54.244464 3967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:54.251559 3967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:05:54.251694 3967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:54.251798 3967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:54.251899 3967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:05:54.252002 3967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:05:54.261912 3967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:54.263258 3967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:54.264613 3967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:54.276799 3972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:54.276706 3975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:54.276717 3973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:54.277465 3967 server_base.cc:1034] running on GCE node
I20260620 03:05:54.278327 3967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:54.279835 3967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:54.281075 3967 hybrid_clock.cc:648] HybridClock initialized: now 1781924754281039 us; error 62 us; skew 500 ppm
I20260620 03:05:54.281425 3967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:54 dist-test-slave-xv68 krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:54.289670 3967 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:05:54.291114 3967 webserver.cc:460] Webserver started at http://127.0.97.62:34599/ using document root <none> and password file <none>
I20260620 03:05:54.291838 3967 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:54.291939 3967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:54.292299 3967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:54.295269 3967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "138d11eb20da40dda00992a965d18f74"
format_stamp: "Formatted at 2026-06-20 03:05:54 on dist-test-slave-xv68"
server_key: "6bfc38b533866a2086243e27df3d5e57"
server_key_iv: "614d109439bb7112ba0a6db50f1f5c2f"
server_key_version: "encryptionkey@0"
I20260620 03:05:54.296092 3967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "138d11eb20da40dda00992a965d18f74"
format_stamp: "Formatted at 2026-06-20 03:05:54 on dist-test-slave-xv68"
server_key: "6bfc38b533866a2086243e27df3d5e57"
server_key_iv: "614d109439bb7112ba0a6db50f1f5c2f"
server_key_version: "encryptionkey@0"
I20260620 03:05:54.301048 3967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:05:54.304225 3982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:54.305450 3967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:05:54.305573 3967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "138d11eb20da40dda00992a965d18f74"
format_stamp: "Formatted at 2026-06-20 03:05:54 on dist-test-slave-xv68"
server_key: "6bfc38b533866a2086243e27df3d5e57"
server_key_iv: "614d109439bb7112ba0a6db50f1f5c2f"
server_key_version: "encryptionkey@0"
I20260620 03:05:54.305727 3967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:54.340785 3967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:54.345108 3967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:54.345331 3967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:54.363730 3967 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42251
I20260620 03:05:54.363750 4033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42251 every 8 connection(s)
I20260620 03:05:54.365025 3967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:05:54.368693 4034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:54.369377 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 3967
I20260620 03:05:54.369608 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:05:54.370412 388 external_mini_cluster.cc:1428] Setting key 41d6129f19ac400aac0e140df517747d
I20260620 03:05:54.382913 4034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74: Bootstrap starting.
Jun 20 03:05:54 dist-test-slave-xv68 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:54.386442 4034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:54.387591 4034 log.cc:824] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:54.390267 4034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74: No bootstrap required, opened a new log
I20260620 03:05:54.393621 4037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:54.372546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33936 (local address 127.0.97.62:42251)
0620 03:05:54.373299 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:54.373310 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:54.373355 (+ 45us) server_negotiation.cc:407] Connection header received
0620 03:05:54.375047 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:54.375072 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:54.375540 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:54.375892 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:54.377654 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:54.378897 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:54.380676 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:54.381338 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:54.381462 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:54.386033 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:54.386041 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:54.386058 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:54.386094 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:54.389171 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:54.389855 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:54.389866 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:54.389872 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:54.390039 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:54.390441 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:54.390445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:54.390447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:54.390973 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:54.391270 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:54.391722 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 03:05:54.392190 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":349,"threads_started":1}
I20260620 03:05:54.398979 4034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138d11eb20da40dda00992a965d18f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42251 } }
I20260620 03:05:54.399335 4034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:54.399454 4034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138d11eb20da40dda00992a965d18f74, State: Initialized, Role: FOLLOWER
I20260620 03:05:54.399986 4034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138d11eb20da40dda00992a965d18f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42251 } }
I20260620 03:05:54.400209 4034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:05:54.400333 4034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:05:54.400496 4034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:54.402511 4034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138d11eb20da40dda00992a965d18f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42251 } }
I20260620 03:05:54.402976 4034 leader_election.cc:302] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 138d11eb20da40dda00992a965d18f74; no voters:
I20260620 03:05:54.403607 4034 leader_election.cc:288] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:05:54.403738 4039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:05:54.404405 4039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74 [term 1 LEADER]: Becoming Leader. State: Replica: 138d11eb20da40dda00992a965d18f74, State: Running, Role: LEADER
I20260620 03:05:54.404901 4039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138d11eb20da40dda00992a965d18f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42251 } }
I20260620 03:05:54.405583 4034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:05:54.408015 4041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 138d11eb20da40dda00992a965d18f74. Latest consensus state: current_term: 1 leader_uuid: "138d11eb20da40dda00992a965d18f74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138d11eb20da40dda00992a965d18f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42251 } } }
I20260620 03:05:54.408001 4040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "138d11eb20da40dda00992a965d18f74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138d11eb20da40dda00992a965d18f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42251 } } }
I20260620 03:05:54.408565 4041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74 [sys.catalog]: This master's current role is: LEADER
I20260620 03:05:54.408591 4040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74 [sys.catalog]: This master's current role is: LEADER
I20260620 03:05:54.412284 4047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:05:54.417222 4047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:05:54.426424 4047 catalog_manager.cc:1347] Generated new cluster ID: 90ed2b2da3a649098dc7e49ad058cb1d
I20260620 03:05:54.426520 4047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:05:54.455866 4047 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:05:54.457214 4047 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:05:54.468418 4047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 138d11eb20da40dda00992a965d18f74: Generated new TSK 0
I20260620 03:05:54.469326 4047 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:05:54.528430 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42251
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:54.854321 4060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:54.854586 4060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:54.854679 4060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:54.861678 4060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:54.861779 4060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:54.861887 4060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:05:54.871207 4060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:54.872610 4060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:54.873803 4060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:54.886941 4065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:54.886870 4068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:54.886904 4066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:54.887873 4060 server_base.cc:1034] running on GCE node
I20260620 03:05:54.888491 4060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:54.889508 4060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:54.890727 4060 hybrid_clock.cc:648] HybridClock initialized: now 1781924754890674 us; error 48 us; skew 500 ppm
I20260620 03:05:54.891028 4060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:54 dist-test-slave-xv68 krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:54.894178 4060 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:05:54.895660 4060 webserver.cc:460] Webserver started at http://127.0.97.1:37411/ using document root <none> and password file <none>
I20260620 03:05:54.896401 4060 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:54.896539 4060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:54.896847 4060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:54.899746 4060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "015525e4cfa5440ab3e290e5e16d27ad"
format_stamp: "Formatted at 2026-06-20 03:05:54 on dist-test-slave-xv68"
server_key: "7f0529890bf0fafcdf4d489cb1081f0e"
server_key_iv: "16798c2d97d7d4e1783befe0e968c949"
server_key_version: "encryptionkey@0"
I20260620 03:05:54.900509 4060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "015525e4cfa5440ab3e290e5e16d27ad"
format_stamp: "Formatted at 2026-06-20 03:05:54 on dist-test-slave-xv68"
server_key: "7f0529890bf0fafcdf4d489cb1081f0e"
server_key_iv: "16798c2d97d7d4e1783befe0e968c949"
server_key_version: "encryptionkey@0"
I20260620 03:05:54.905647 4060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 03:05:54.909189 4075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:54.910725 4060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:05:54.910882 4060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "015525e4cfa5440ab3e290e5e16d27ad"
format_stamp: "Formatted at 2026-06-20 03:05:54 on dist-test-slave-xv68"
server_key: "7f0529890bf0fafcdf4d489cb1081f0e"
server_key_iv: "16798c2d97d7d4e1783befe0e968c949"
server_key_version: "encryptionkey@0"
I20260620 03:05:54.911083 4060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:54.926648 4060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:54.930691 4060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:54.930984 4060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:54.931970 4060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:05:54.933369 4060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:54.933483 4060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:54.933632 4060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:54.933703 4060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:54.966588 4060 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44317
I20260620 03:05:54.966624 4187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44317 every 8 connection(s)
I20260620 03:05:54.967919 4060 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:05:54.976972 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 4060
I20260620 03:05:54.977254 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:05:54.978226 388 external_mini_cluster.cc:1428] Setting key 552f03a321dad0d6f56762b69b223524
Jun 20 03:05:54 dist-test-slave-xv68 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:54.994972 4191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:54.971019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42251 (local address 127.0.97.1:42181)
0620 03:05:54.971794 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:54.971836 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:54.974074 (+ 2238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:54.974929 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:54.974946 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:54.975891 (+ 945us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:54.976600 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:54.976834 (+ 234us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:54.978472 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:54.978478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:54.980356 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:54.980367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:54.981524 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:54.981530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:54.981739 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:54.982612 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:54.982636 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:54.985623 (+ 2987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:54.989902 (+ 4279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:54.989914 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:54.989934 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:54.990543 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:54.990990 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:54.990998 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:54.991004 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:54.991247 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:54.992092 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:54.992102 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:54.992555 (+ 453us) client_negotiation.cc:769] Sending connection context
0620 03:05:54.992844 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 03:05:54.993439 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":356,"threads_started":1}
I20260620 03:05:54.996256 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:54.970421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42181 (local address 127.0.97.62:42251)
0620 03:05:54.970849 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:54.970853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:54.971946 (+ 1093us) server_negotiation.cc:407] Connection header received
0620 03:05:54.974344 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:54.974349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:54.974439 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:54.974560 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:54.977046 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:54.978276 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:54.980491 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:54.981145 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:54.981280 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:54.985899 (+ 4619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:54.985907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:54.985913 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:54.985957 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:54.989740 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:54.990704 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:54.990712 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:54.990731 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:54.990870 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:54.991445 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:54.991452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:54.991458 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:54.991850 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:54.991991 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:54.995541 (+ 3550us) server_negotiation.cc:299] Negotiation successful
0620 03:05:54.995786 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":245,"threads_started":1}
I20260620 03:05:54.998981 4188 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42251
I20260620 03:05:54.999377 4188 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:55.000382 4188 heartbeater.cc:511] Master 127.0.97.62:42251 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 03:05:55.003260 3999 ts_manager.cc:194] Registered new tserver with Master: 015525e4cfa5440ab3e290e5e16d27ad (127.0.97.1:44317)
I20260620 03:05:55.004832 3999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:05:55.030756 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42251
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:55.346429 4194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:55.346748 4194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:55.346920 4194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:55.354175 4194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:55.354324 4194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:55.354511 4194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:05:55.364423 4194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:55.365613 4194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:55.366845 4194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:55.379819 4199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:55.379966 4200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:55.379925 4202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:55.380926 4194 server_base.cc:1034] running on GCE node
I20260620 03:05:55.381527 4194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:55.382464 4194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:55.383682 4194 hybrid_clock.cc:648] HybridClock initialized: now 1781924755383618 us; error 59 us; skew 500 ppm
I20260620 03:05:55.383986 4194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:55 dist-test-slave-xv68 krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:55.386983 4194 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:05:55.388465 4194 webserver.cc:460] Webserver started at http://127.0.97.2:46399/ using document root <none> and password file <none>
I20260620 03:05:55.389078 4194 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:55.389225 4194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:55.389534 4194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:55.392410 4194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "16c5cfb893bb46d192ff015f70efb855"
format_stamp: "Formatted at 2026-06-20 03:05:55 on dist-test-slave-xv68"
server_key: "a1440799812eb9cc7ab69e7af36bb4b7"
server_key_iv: "22cbd064adf758aa584c65edba167eca"
server_key_version: "encryptionkey@0"
I20260620 03:05:55.393157 4194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "16c5cfb893bb46d192ff015f70efb855"
format_stamp: "Formatted at 2026-06-20 03:05:55 on dist-test-slave-xv68"
server_key: "a1440799812eb9cc7ab69e7af36bb4b7"
server_key_iv: "22cbd064adf758aa584c65edba167eca"
server_key_version: "encryptionkey@0"
I20260620 03:05:55.400128 4194 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 03:05:55.404502 4209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:55.405812 4194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260620 03:05:55.405980 4194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "16c5cfb893bb46d192ff015f70efb855"
format_stamp: "Formatted at 2026-06-20 03:05:55 on dist-test-slave-xv68"
server_key: "a1440799812eb9cc7ab69e7af36bb4b7"
server_key_iv: "22cbd064adf758aa584c65edba167eca"
server_key_version: "encryptionkey@0"
I20260620 03:05:55.406203 4194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:55.420840 4194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:55.424602 4194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:55.424880 4194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:55.425792 4194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:05:55.427045 4194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:55.427114 4194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:55.427280 4194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:55.427393 4194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 03:05:55.459815 4194 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36817
I20260620 03:05:55.459857 4321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36817 every 8 connection(s)
I20260620 03:05:55.461057 4194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:05:55.470811 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 4194
I20260620 03:05:55.471020 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:05:55.471534 388 external_mini_cluster.cc:1428] Setting key 8b6e2db3ab0493e6509cb450d9419e9d
Jun 20 03:05:55 dist-test-slave-xv68 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:55.480783 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:55.463623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45041 (local address 127.0.97.62:42251)
0620 03:05:55.463815 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:55.463819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:55.464910 (+ 1091us) server_negotiation.cc:407] Connection header received
0620 03:05:55.466637 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:55.466642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:55.466729 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:55.466844 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:55.468286 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:55.469039 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:55.470762 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:55.471285 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:55.471438 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:55.475284 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:55.475329 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:55.475335 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:55.475372 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:55.477876 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:55.478558 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:55.478565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:55.478573 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:55.478689 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:55.479084 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:55.479091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:55.479094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:55.479429 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:55.479581 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:55.480231 (+ 650us) server_negotiation.cc:299] Negotiation successful
0620 03:05:55.480442 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 03:05:55.481767 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:55.464031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42251 (local address 127.0.97.2:45041)
0620 03:05:55.464750 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:55.464793 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:55.466377 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:55.467070 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:55.467077 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:55.467580 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:55.468118 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:55.468133 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:55.469187 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:55.469201 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:55.470531 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:55.470546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:55.471548 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:55.471559 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:55.471853 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:55.472700 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:55.472757 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:55.475078 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:55.478018 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:55.478026 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:55.478042 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:55.478437 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:55.478794 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:55.478800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:55.478803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:55.478979 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:55.479555 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:55.479563 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:55.479895 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 03:05:55.480164 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 03:05:55.480674 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":300,"threads_started":1}
I20260620 03:05:55.483433 4322 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42251
I20260620 03:05:55.483736 4322 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:55.484422 4322 heartbeater.cc:511] Master 127.0.97.62:42251 requested a full tablet report, sending...
I20260620 03:05:55.486011 3999 ts_manager.cc:194] Registered new tserver with Master: 16c5cfb893bb46d192ff015f70efb855 (127.0.97.2:36817)
I20260620 03:05:55.487517 3999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45041
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:05:55.515383 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42251
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:55.832798 4327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:55.833070 4327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:55.833248 4327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:55.840576 4327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:55.840677 4327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:55.840886 4327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:05:55.850414 4327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:55.851770 4327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:55.853072 4327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:55.865805 4332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:55.865859 4333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:55.865824 4335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:55.866645 4327 server_base.cc:1034] running on GCE node
I20260620 03:05:55.867283 4327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:55.868204 4327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:55.869422 4327 hybrid_clock.cc:648] HybridClock initialized: now 1781924755869377 us; error 58 us; skew 500 ppm
I20260620 03:05:55.869724 4327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:55 dist-test-slave-xv68 krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:55.872781 4327 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:05:55.874322 4327 webserver.cc:460] Webserver started at http://127.0.97.3:34683/ using document root <none> and password file <none>
I20260620 03:05:55.874938 4327 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:55.875034 4327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:55.875280 4327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:55.878098 4327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "c94fdd9dfe9d4dbe9c42f66ba874d572"
format_stamp: "Formatted at 2026-06-20 03:05:55 on dist-test-slave-xv68"
server_key: "54943d36f838990a3cd1d7c2ccb5e544"
server_key_iv: "94b1a471d669690f75dd2cd1b0caab96"
server_key_version: "encryptionkey@0"
I20260620 03:05:55.878830 4327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c94fdd9dfe9d4dbe9c42f66ba874d572"
format_stamp: "Formatted at 2026-06-20 03:05:55 on dist-test-slave-xv68"
server_key: "54943d36f838990a3cd1d7c2ccb5e544"
server_key_iv: "94b1a471d669690f75dd2cd1b0caab96"
server_key_version: "encryptionkey@0"
I20260620 03:05:55.883911 4327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 03:05:55.887135 4342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:55.888484 4327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:05:55.888613 4327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "c94fdd9dfe9d4dbe9c42f66ba874d572"
format_stamp: "Formatted at 2026-06-20 03:05:55 on dist-test-slave-xv68"
server_key: "54943d36f838990a3cd1d7c2ccb5e544"
server_key_iv: "94b1a471d669690f75dd2cd1b0caab96"
server_key_version: "encryptionkey@0"
I20260620 03:05:55.888775 4327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:55.903580 4327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:55.907584 4327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:55.907809 4327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:55.908651 4327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:05:55.910049 4327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:55.910204 4327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 03:05:55.910336 4327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:55.910405 4327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:55.943419 4327 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40461
I20260620 03:05:55.943442 4454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40461 every 8 connection(s)
I20260620 03:05:55.944698 4327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:05:55.952386 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 4327
I20260620 03:05:55.952589 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:05:55.952987 388 external_mini_cluster.cc:1428] Setting key 7ebe171cd212b32016fbfde8e69fcf6e
Jun 20 03:05:55 dist-test-slave-xv68 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:55.964175 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:55.947177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59313 (local address 127.0.97.62:42251)
0620 03:05:55.947393 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:55.947397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:55.948595 (+ 1198us) server_negotiation.cc:407] Connection header received
0620 03:05:55.950420 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:55.950426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:55.950517 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:55.950628 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:55.952189 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:55.953046 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:55.954798 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:55.955362 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:55.955466 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:55.958853 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:55.958858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:55.958861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:55.958894 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:55.961483 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:55.962184 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:55.962187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:55.962190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:55.962308 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:55.962737 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:55.962746 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:55.962748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:55.962992 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:55.963097 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:55.963792 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 03:05:55.963941 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 03:05:55.965148 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:55.947711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42251 (local address 127.0.97.3:59313)
0620 03:05:55.948423 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:55.948466 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:55.950145 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:55.950866 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:55.950874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:55.951450 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:55.952006 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:55.952036 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:55.953211 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:55.953215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:55.954656 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:55.954677 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:55.955578 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:55.955585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:55.955904 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:55.956528 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:55.956549 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:55.958615 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:55.961677 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:55.961683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:55.961695 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:55.962084 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:55.962402 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:55.962405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:55.962407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:55.962632 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:55.963092 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:55.963097 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:55.963478 (+ 381us) client_negotiation.cc:769] Sending connection context
0620 03:05:55.963702 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 03:05:55.964068 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":295,"threads_started":1}
I20260620 03:05:55.966727 4455 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42251
I20260620 03:05:55.967069 4455 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:55.967962 4455 heartbeater.cc:511] Master 127.0.97.62:42251 requested a full tablet report, sending...
I20260620 03:05:55.969424 3999 ts_manager.cc:194] Registered new tserver with Master: c94fdd9dfe9d4dbe9c42f66ba874d572 (127.0.97.3:40461)
I20260620 03:05:55.970316 3999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59313
I20260620 03:05:55.980526 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 03:05:56 dist-test-slave-xv68 krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260620 03:05:56.009922 4188 heartbeater.cc:503] Master 127.0.97.62:42251 was elected leader, sending a full tablet report...
Jun 20 03:05:56 dist-test-slave-xv68 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:05:56.027910 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:56.013326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58482 (local address 127.0.97.1:44317)
0620 03:05:56.013911 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:56.013918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:56.013943 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:05:56.014018 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:56.014024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:56.014227 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:56.014356 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:56.015709 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:56.016774 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:56.018069 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:56.018623 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:56.018717 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:56.022509 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:56.022520 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:56.022534 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:56.022566 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:56.025366 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:56.025961 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:56.025967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:56.025970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:56.026078 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:56.026471 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:56.026475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:56.026477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:56.026787 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:56.027006 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:56.027261 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 03:05:56.027496 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":316,"threads_started":1}
Jun 20 03:05:56 dist-test-slave-xv68 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:56.044584 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:56.031965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33948 (local address 127.0.97.62:42251)
0620 03:05:56.032185 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:56.032191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:56.032370 (+ 179us) server_negotiation.cc:407] Connection header received
0620 03:05:56.032463 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:56.032467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:56.032539 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:56.032625 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:56.033742 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:56.034474 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:56.035714 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:56.036122 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:56.036225 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:56.039940 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:56.039945 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:56.039948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:56.039978 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:56.042345 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:56.042898 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:56.042902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:56.042904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:56.043018 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:56.043412 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:56.043415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:56.043417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:56.043706 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:56.043882 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:56.044116 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 03:05:56.044270 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 03:05:56.045128 3999 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33948
I20260620 03:05:56.046067 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 4060
I20260620 03:05:56.057539 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:56.046887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58496 (local address 127.0.97.1:44317)
0620 03:05:56.047103 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:56.047112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:56.047394 (+ 282us) server_negotiation.cc:407] Connection header received
0620 03:05:56.047516 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:56.047519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:56.047577 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:56.047650 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:56.048720 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:56.049367 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:56.050497 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:56.050889 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:56.050976 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:56.053029 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:56.053034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:56.053037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:56.053064 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:56.055614 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:56.056119 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:56.056123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:56.056125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:56.056228 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:56.056514 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:56.056517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:56.056519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:56.056723 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:56.056884 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:56.057085 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 03:05:56.057224 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:05:56.058080 4183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:05:56.140642 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 4060
I20260620 03:05:56.157169 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 4194
Jun 20 03:05:56 dist-test-slave-xv68 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:05:56.171645 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:56.157993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47298 (local address 127.0.97.2:36817)
0620 03:05:56.158514 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:56.158521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:56.158546 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:05:56.158616 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:56.158630 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:56.158849 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:56.158978 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:56.160231 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:56.161221 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:56.162370 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:56.162901 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:56.162991 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:56.166798 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:56.166808 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:56.166821 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:56.166850 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:56.169406 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:56.169952 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:56.169958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:56.169960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:56.170070 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:56.170386 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:56.170389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:56.170391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:56.170694 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:56.170897 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:56.171116 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 03:05:56.171363 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":273,"threads_started":1}
I20260620 03:05:56.172819 4317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:05:56.253477 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 4194
I20260620 03:05:56.269253 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 4327
Jun 20 03:05:56 dist-test-slave-xv68 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:05:56.285682 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:56.270336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54204 (local address 127.0.97.3:40461)
0620 03:05:56.270945 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:56.270952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:56.270979 (+ 27us) server_negotiation.cc:407] Connection header received
0620 03:05:56.271034 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:56.271040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:56.271239 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:56.271449 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:56.272677 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:56.273688 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:56.275220 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:56.275858 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:56.275963 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:56.280121 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:56.280133 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:56.280147 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:56.280180 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:56.283026 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:56.283770 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:56.283777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:56.283780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:56.283895 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:56.284268 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:56.284271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:56.284273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:56.284598 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:56.284823 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:56.285065 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 03:05:56.285269 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":320,"threads_started":1}
I20260620 03:05:56.287019 4450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:05:56.369095 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 4327
I20260620 03:05:56.385746 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 3967
I20260620 03:05:56.386902 4029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:05:56.488250 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 3967
2026-06-20T03:05:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3833 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:05:56 dist-test-slave-xv68 krb5kdc[4474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:05:56 dist-test-slave-xv68 krb5kdc[4474](info): set up 2 sockets
Jun 20 03:05:56 dist-test-slave-xv68 krb5kdc[4474](info): commencing operation
krb5kdc: starting...
W20260620 03:05:57.575670 388 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:05:57 dist-test-slave-xv68 krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:05:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:05:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:05:57.766364 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34621
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:58.083781 4487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:58.084158 4487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:58.084326 4487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:58.091475 4487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:05:58.091624 4487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:58.091737 4487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:58.091841 4487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:05:58.091934 4487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:05:58.101678 4487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:58.103017 4487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:58.104337 4487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:58.115526 4492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:58.115562 4495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:58.115654 4493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:58.116340 4487 server_base.cc:1034] running on GCE node
I20260620 03:05:58.117097 4487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:58.118523 4487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:58.119755 4487 hybrid_clock.cc:648] HybridClock initialized: now 1781924758119710 us; error 35 us; skew 500 ppm
I20260620 03:05:58.120087 4487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:58 dist-test-slave-xv68 krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:58.123085 4487 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:05:58.124285 4487 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:58.124369 4487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:58.124614 4487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:58.127518 4487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "4851aa02718e4699ad07f575d3049b07"
format_stamp: "Formatted at 2026-06-20 03:05:58 on dist-test-slave-xv68"
server_key: "d4fd16ca1f80335f1e6f8d46fad788c0"
server_key_iv: "27557894ddcd26f27b9d1e403b57bffa"
server_key_version: "encryptionkey@0"
I20260620 03:05:58.128295 4487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "4851aa02718e4699ad07f575d3049b07"
format_stamp: "Formatted at 2026-06-20 03:05:58 on dist-test-slave-xv68"
server_key: "d4fd16ca1f80335f1e6f8d46fad788c0"
server_key_iv: "27557894ddcd26f27b9d1e403b57bffa"
server_key_version: "encryptionkey@0"
I20260620 03:05:58.133165 4487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:05:58.136526 4501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:58.137850 4487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:05:58.137988 4487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "4851aa02718e4699ad07f575d3049b07"
format_stamp: "Formatted at 2026-06-20 03:05:58 on dist-test-slave-xv68"
server_key: "d4fd16ca1f80335f1e6f8d46fad788c0"
server_key_iv: "27557894ddcd26f27b9d1e403b57bffa"
server_key_version: "encryptionkey@0"
I20260620 03:05:58.138173 4487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:58.151887 4487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:58.155951 4487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:58.156191 4487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:58.175519 4487 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34621
I20260620 03:05:58.175490 4552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34621 every 8 connection(s)
I20260620 03:05:58.176476 4487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:05:58.180408 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:58.183465 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 4487
I20260620 03:05:58.183686 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:05:58.184418 388 external_mini_cluster.cc:1428] Setting key fed73ce035aa19753445a76cd0fda2ea
I20260620 03:05:58.193135 4553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07: Bootstrap starting.
I20260620 03:05:58.196825 4553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:58.198114 4553 log.cc:824] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07: Log is configured to *not* fsync() on all Append() calls
Jun 20 03:05:58 dist-test-slave-xv68 krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:58.200956 4553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07: No bootstrap required, opened a new log
I20260620 03:05:58.208268 4556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:58.186585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48014 (local address 127.0.97.62:34621)
0620 03:05:58.187434 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:58.187447 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:58.187488 (+ 41us) server_negotiation.cc:407] Connection header received
0620 03:05:58.189080 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:58.189103 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:58.189512 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:58.189840 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:58.191651 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:58.192897 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:58.194695 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:58.195560 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:58.195715 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:58.200504 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:58.200512 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:58.200526 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:58.200560 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:58.203661 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:58.204344 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:58.204351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:58.204357 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:58.204512 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:58.204965 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:58.204973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:58.204981 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:58.205739 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:58.206044 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:58.206448 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 03:05:58.206888 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":422,"threads_started":1}
I20260620 03:05:58.209869 4553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4851aa02718e4699ad07f575d3049b07" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34621 } }
I20260620 03:05:58.210213 4553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:58.210348 4553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4851aa02718e4699ad07f575d3049b07, State: Initialized, Role: FOLLOWER
I20260620 03:05:58.210976 4553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4851aa02718e4699ad07f575d3049b07" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34621 } }
I20260620 03:05:58.211203 4553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:05:58.211377 4553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:05:58.211565 4553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:05:58.213699 4553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4851aa02718e4699ad07f575d3049b07" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34621 } }
I20260620 03:05:58.214175 4553 leader_election.cc:302] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4851aa02718e4699ad07f575d3049b07; no voters:
I20260620 03:05:58.214742 4553 leader_election.cc:288] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:05:58.214876 4558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:05:58.215651 4558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07 [term 1 LEADER]: Becoming Leader. State: Replica: 4851aa02718e4699ad07f575d3049b07, State: Running, Role: LEADER
I20260620 03:05:58.216135 4558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4851aa02718e4699ad07f575d3049b07" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34621 } }
I20260620 03:05:58.216768 4553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:05:58.218950 4559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4851aa02718e4699ad07f575d3049b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4851aa02718e4699ad07f575d3049b07" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34621 } } }
I20260620 03:05:58.219367 4559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07 [sys.catalog]: This master's current role is: LEADER
I20260620 03:05:58.219601 4560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4851aa02718e4699ad07f575d3049b07. Latest consensus state: current_term: 1 leader_uuid: "4851aa02718e4699ad07f575d3049b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4851aa02718e4699ad07f575d3049b07" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34621 } } }
I20260620 03:05:58.219981 4560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07 [sys.catalog]: This master's current role is: LEADER
I20260620 03:05:58.226964 4564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:05:58.232337 4564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:05:58.240854 4564 catalog_manager.cc:1347] Generated new cluster ID: bc098371cd4c4fbc90ffbbbf96aa3c4d
I20260620 03:05:58.240967 4564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:05:58.251169 4564 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:05:58.252439 4564 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:05:58.261583 4564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07: Generated new TSK 0
I20260620 03:05:58.262538 4564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:05:58.318930 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34621
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:58.637346 4579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:58.637648 4579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:58.637811 4579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:58.645036 4579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:58.645182 4579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:58.645365 4579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:05:58.654680 4579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:58.655944 4579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:58.657205 4579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:58.669387 4584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:58.669349 4587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:58.669880 4585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:58.670347 4579 server_base.cc:1034] running on GCE node
I20260620 03:05:58.670888 4579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:58.671833 4579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:58.673027 4579 hybrid_clock.cc:648] HybridClock initialized: now 1781924758672959 us; error 61 us; skew 500 ppm
I20260620 03:05:58.673343 4579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:58 dist-test-slave-xv68 krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:58.676440 4579 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:05:58.677522 4579 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:58.677609 4579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:58.677850 4579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:58.680864 4579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "8cde57d548384c2585da6d7fe9342a4a"
format_stamp: "Formatted at 2026-06-20 03:05:58 on dist-test-slave-xv68"
server_key: "1acd190161ec9f857b30cdb3fcd46c1d"
server_key_iv: "c1a3aabb2c6955c3bec7491788b54e01"
server_key_version: "encryptionkey@0"
I20260620 03:05:58.681645 4579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8cde57d548384c2585da6d7fe9342a4a"
format_stamp: "Formatted at 2026-06-20 03:05:58 on dist-test-slave-xv68"
server_key: "1acd190161ec9f857b30cdb3fcd46c1d"
server_key_iv: "c1a3aabb2c6955c3bec7491788b54e01"
server_key_version: "encryptionkey@0"
I20260620 03:05:58.686761 4579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260620 03:05:58.690318 4593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:58.691823 4579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 03:05:58.691989 4579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "8cde57d548384c2585da6d7fe9342a4a"
format_stamp: "Formatted at 2026-06-20 03:05:58 on dist-test-slave-xv68"
server_key: "1acd190161ec9f857b30cdb3fcd46c1d"
server_key_iv: "c1a3aabb2c6955c3bec7491788b54e01"
server_key_version: "encryptionkey@0"
I20260620 03:05:58.692224 4579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:58.719950 4579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:58.724129 4579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:58.724416 4579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:58.725466 4579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:05:58.726969 4579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:58.727059 4579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:58.727226 4579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:58.727345 4579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 03:05:58.760223 4579 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46615
I20260620 03:05:58.760327 4705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46615 every 8 connection(s)
I20260620 03:05:58.761207 4579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:05:58.768743 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 4579
I20260620 03:05:58.768958 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:05:58.769657 388 external_mini_cluster.cc:1428] Setting key 30e7332b4bc6b5af511ae799d6fe4637
Jun 20 03:05:58 dist-test-slave-xv68 krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:58.782407 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:58.763695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43693 (local address 127.0.97.62:34621)
0620 03:05:58.764183 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:58.764188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:58.765094 (+ 906us) server_negotiation.cc:407] Connection header received
0620 03:05:58.766782 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:58.766786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:58.766855 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:58.766949 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:58.768429 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:58.769572 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:58.771081 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:58.771622 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:58.771722 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:58.775830 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:58.775839 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:58.775846 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:58.775894 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:58.778770 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:58.779803 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:58.779808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:58.779810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:58.779927 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:58.780518 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:58.780522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:58.780524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:58.780837 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:58.780953 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:58.781884 (+ 931us) server_negotiation.cc:299] Negotiation successful
0620 03:05:58.782062 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":233,"threads_started":1}
I20260620 03:05:58.783932 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:58.764281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34621 (local address 127.0.97.1:43693)
0620 03:05:58.764942 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:58.764980 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:58.766532 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:58.767140 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:58.767147 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:58.767737 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:58.768269 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:58.768285 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:58.769733 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:58.769740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:58.770946 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:58.770957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:58.771848 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:58.771856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:58.772148 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:58.772873 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:58.772913 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:58.775605 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:58.778916 (+ 3311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:58.778928 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:58.778948 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:58.779688 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:58.780036 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:58.780043 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:58.780050 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:58.780315 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:58.780977 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:58.780989 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:58.781478 (+ 489us) client_negotiation.cc:769] Sending connection context
0620 03:05:58.781791 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 03:05:58.782388 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":268,"threads_started":1}
I20260620 03:05:58.785617 4706 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34621
I20260620 03:05:58.785952 4706 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:58.786693 4706 heartbeater.cc:511] Master 127.0.97.62:34621 requested a full tablet report, sending...
I20260620 03:05:58.788839 4518 ts_manager.cc:194] Registered new tserver with Master: 8cde57d548384c2585da6d7fe9342a4a (127.0.97.1:46615)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 03:05:58.790342 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:05:58.814872 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34621
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:59.133239 4712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:59.133574 4712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:59.133751 4712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:59.140934 4712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:59.141094 4712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:59.141288 4712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:05:59.150946 4712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:59.152295 4712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:59.153570 4712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:59.166244 4718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:59.166265 4720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:59.166517 4717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:59.167083 4712 server_base.cc:1034] running on GCE node
I20260620 03:05:59.167657 4712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:59.168510 4712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:59.169703 4712 hybrid_clock.cc:648] HybridClock initialized: now 1781924759169665 us; error 31 us; skew 500 ppm
I20260620 03:05:59.170022 4712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:59 dist-test-slave-xv68 krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:59.173103 4712 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:05:59.174137 4712 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:59.174219 4712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:59.174466 4712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:59.177277 4712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "9c900c6658274e76be8d786920ee084c"
format_stamp: "Formatted at 2026-06-20 03:05:59 on dist-test-slave-xv68"
server_key: "1bb02605fa99085a86db21ea999489c7"
server_key_iv: "fba9dc5acd1ef0a91eabc8e08a519874"
server_key_version: "encryptionkey@0"
I20260620 03:05:59.178089 4712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9c900c6658274e76be8d786920ee084c"
format_stamp: "Formatted at 2026-06-20 03:05:59 on dist-test-slave-xv68"
server_key: "1bb02605fa99085a86db21ea999489c7"
server_key_iv: "fba9dc5acd1ef0a91eabc8e08a519874"
server_key_version: "encryptionkey@0"
I20260620 03:05:59.183517 4712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 03:05:59.187103 4726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:59.188499 4712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:05:59.188676 4712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "9c900c6658274e76be8d786920ee084c"
format_stamp: "Formatted at 2026-06-20 03:05:59 on dist-test-slave-xv68"
server_key: "1bb02605fa99085a86db21ea999489c7"
server_key_iv: "fba9dc5acd1ef0a91eabc8e08a519874"
server_key_version: "encryptionkey@0"
I20260620 03:05:59.188905 4712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:59.223703 4712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:59.228157 4712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:59.228444 4712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:59.229434 4712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:05:59.230767 4712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:59.230887 4712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:59.231022 4712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:59.231109 4712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:59.265324 4712 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44573
I20260620 03:05:59.265407 4838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44573 every 8 connection(s)
I20260620 03:05:59.266191 4712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:05:59.268251 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 4712
I20260620 03:05:59.268442 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:05:59.268967 388 external_mini_cluster.cc:1428] Setting key 319a0c2fd0b32270acf10bc0b3bea3ed
Jun 20 03:05:59 dist-test-slave-xv68 krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 03:05:59.289430 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:59.268829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35229 (local address 127.0.97.62:34621)
0620 03:05:59.269028 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:59.269034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:59.270435 (+ 1401us) server_negotiation.cc:407] Connection header received
0620 03:05:59.273170 (+ 2735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:59.273205 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:59.273270 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:59.273353 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:59.275507 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.276253 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.278177 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.278963 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.279112 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:59.282716 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:59.282725 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:59.282733 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:59.282779 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:59.286126 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:59.287070 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:59.287076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:59.287086 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:59.287219 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:59.287661 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:59.287673 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:59.287675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:59.287927 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:59.288081 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:59.288811 (+ 730us) server_negotiation.cc:299] Negotiation successful
0620 03:05:59.289053 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 03:05:59.290431 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:59.269376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34621 (local address 127.0.97.2:35229)
0620 03:05:59.270261 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:59.270312 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:59.272863 (+ 2551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:59.273620 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:59.273635 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:59.274432 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:59.275258 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:59.275322 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.276408 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.276416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:59.278008 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:59.278020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.279179 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.279184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:59.279471 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:59.280182 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:59.280215 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:59.282464 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:59.286432 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:59.286448 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:59.286464 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:59.286935 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:59.287362 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:59.287372 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:59.287374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:59.287541 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:59.288051 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:59.288059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:59.288427 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 03:05:59.288717 (+ 290us) client_negotiation.cc:240] Negotiation successful
0620 03:05:59.289265 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":384,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 03:05:59.292099 4839 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34621
I20260620 03:05:59.292440 4839 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:59.293251 4839 heartbeater.cc:511] Master 127.0.97.62:34621 requested a full tablet report, sending...
I20260620 03:05:59.294759 4518 ts_manager.cc:194] Registered new tserver with Master: 9c900c6658274e76be8d786920ee084c (127.0.97.2:44573)
I20260620 03:05:59.295688 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:05:59.315225 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34621
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:05:59.634655 4844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:05:59.634929 4844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:05:59.635027 4844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:05:59.642217 4844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:05:59.642330 4844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:05:59.642446 4844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:05:59.651921 4844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:05:59.653187 4844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:05:59.654460 4844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:05:59.667433 4850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:59.667537 4849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:05:59.667516 4852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:05:59.672773 4844 server_base.cc:1034] running on GCE node
I20260620 03:05:59.673308 4844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:05:59.674144 4844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:05:59.675384 4844 hybrid_clock.cc:648] HybridClock initialized: now 1781924759675341 us; error 26 us; skew 500 ppm
I20260620 03:05:59.675698 4844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:05:59 dist-test-slave-xv68 krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:05:59.682454 4844 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:05:59.683800 4844 fs_manager.cc:359] Metadata directory not provided
I20260620 03:05:59.683935 4844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:05:59.684242 4844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:05:59.687078 4844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "1927d27f3e2f4698a50a4f7545328f38"
format_stamp: "Formatted at 2026-06-20 03:05:59 on dist-test-slave-xv68"
server_key: "d3aa81e23da1ecd3252ff51863154870"
server_key_iv: "853bf21b0e5a896cfd6130141e31df53"
server_key_version: "encryptionkey@0"
I20260620 03:05:59.688139 4844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1927d27f3e2f4698a50a4f7545328f38"
format_stamp: "Formatted at 2026-06-20 03:05:59 on dist-test-slave-xv68"
server_key: "d3aa81e23da1ecd3252ff51863154870"
server_key_iv: "853bf21b0e5a896cfd6130141e31df53"
server_key_version: "encryptionkey@0"
I20260620 03:05:59.693671 4844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:05:59.697197 4858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:59.698603 4844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 03:05:59.698779 4844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "1927d27f3e2f4698a50a4f7545328f38"
format_stamp: "Formatted at 2026-06-20 03:05:59 on dist-test-slave-xv68"
server_key: "d3aa81e23da1ecd3252ff51863154870"
server_key_iv: "853bf21b0e5a896cfd6130141e31df53"
server_key_version: "encryptionkey@0"
I20260620 03:05:59.698994 4844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:05:59.715282 4844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:05:59.719421 4844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:05:59.719684 4844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:05:59.720582 4844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:05:59.721868 4844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:05:59.721994 4844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:59.722114 4844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:05:59.722205 4844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:05:59.755080 4844 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46775
I20260620 03:05:59.755165 4970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46775 every 8 connection(s)
I20260620 03:05:59.755946 4844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:05:59.764325 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 4844
I20260620 03:05:59.764540 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:05:59.764938 388 external_mini_cluster.cc:1428] Setting key f980abc8178bc6f90f05df32493f625a
Jun 20 03:05:59 dist-test-slave-xv68 krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:05:59.774937 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:59.758358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51439 (local address 127.0.97.62:34621)
0620 03:05:59.758535 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:59.758540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:59.759654 (+ 1114us) server_negotiation.cc:407] Connection header received
0620 03:05:59.761474 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:59.761481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:59.761574 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:59.761688 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:59.763339 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.764109 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.765821 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.766248 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.766380 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:59.769707 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:59.769725 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:59.769728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:59.769758 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:59.772117 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:59.772797 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:59.772803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:59.772807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:59.772929 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:59.773327 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:59.773333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:59.773337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:59.773595 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:59.773747 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:59.774382 (+ 635us) server_negotiation.cc:299] Negotiation successful
0620 03:05:59.774589 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:05:59.775933 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:59.758748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34621 (local address 127.0.97.3:51439)
0620 03:05:59.759486 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:59.759530 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:59.761205 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:59.761905 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:59.761913 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:59.762406 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:05:59.763121 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:59.763153 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.764267 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.764272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:59.765670 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:59.765681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.766462 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.766470 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:59.766691 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:59.767405 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:05:59.767426 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:05:59.769510 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:05:59.772257 (+ 2747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:59.772265 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:59.772281 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:59.772670 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:59.773036 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:05:59.773042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:05:59.773045 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:05:59.773212 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:05:59.773725 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:05:59.773733 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:05:59.774032 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 03:05:59.774295 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 03:05:59.774768 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":277,"threads_started":1}
I20260620 03:05:59.777283 4971 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34621
I20260620 03:05:59.777594 4971 heartbeater.cc:464] Registering TS with master...
I20260620 03:05:59.778249 4971 heartbeater.cc:511] Master 127.0.97.62:34621 requested a full tablet report, sending...
I20260620 03:05:59.779800 4518 ts_manager.cc:194] Registered new tserver with Master: 1927d27f3e2f4698a50a4f7545328f38 (127.0.97.3:46775)
I20260620 03:05:59.780627 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51439
I20260620 03:05:59.792392 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:05:59.793485 4706 heartbeater.cc:503] Master 127.0.97.62:34621 was elected leader, sending a full tablet report...
I20260620 03:05:59.808893 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:59.797130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48018 (local address 127.0.97.62:34621)
0620 03:05:59.797343 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:59.797351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:59.797842 (+ 491us) server_negotiation.cc:407] Connection header received
0620 03:05:59.797968 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:59.797971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:59.798028 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:59.798111 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:05:59.799505 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.800213 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.801391 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.801788 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.801885 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:59.804062 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:05:59.804066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:05:59.804070 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:05:59.804098 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:05:59.806551 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:59.807120 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:59.807123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:59.807125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:59.807227 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:05:59.807636 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:05:59.807639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:05:59.807641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:05:59.807928 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:05:59.808098 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:59.808405 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 03:05:59.808552 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 03:05:59.813169 4518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:05:59.820047 4518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:05:59.854333 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:59.833821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46615 (local address 127.0.0.1:45500)
0620 03:05:59.845873 (+ 12052us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:59.845885 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:59.845952 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:59.847130 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:59.847137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:59.847179 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:59.847604 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:59.847616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.849230 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.849236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:59.851409 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:59.851426 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.853220 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.853225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:59.853426 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:59.853478 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 03:05:59.853598 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 03:05:59.853792 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11961,"mutex_wait_us":32,"thread_start_us":11453,"threads_started":1}
I20260620 03:05:59.855109 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:59.834133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.0.97.1:46615)
0620 03:05:59.846372 (+ 12239us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:59.846379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:59.846404 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:05:59.846475 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:59.846481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:59.846817 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:59.847065 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:59.847817 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.849081 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.851596 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.852682 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.852783 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:59.853019 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:59.854614 (+ 1595us) server_negotiation.cc:299] Negotiation successful
0620 03:05:59.854790 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12099,"thread_start_us":11958,"threads_started":1}
I20260620 03:05:59.856276 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:59.832475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44573 (local address 127.0.0.1:50920)
0620 03:05:59.844392 (+ 11917us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:59.844444 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:59.844633 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:59.848482 (+ 3849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:59.848486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:59.848507 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:59.848893 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:59.848904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.850208 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.850216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:59.852865 (+ 2649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:59.852879 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.855429 (+ 2550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.855438 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:59.855601 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:59.855634 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 03:05:59.855726 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 03:05:59.855862 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11635,"mutex_wait_us":150,"thread_start_us":350,"threads_started":1}
I20260620 03:05:59.857074 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:59.832491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46775 (local address 127.0.0.1:60662)
0620 03:05:59.845519 (+ 13028us) negotiation.cc:107] Waiting for socket to connect
0620 03:05:59.845534 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:05:59.845618 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:05:59.847675 (+ 2057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:05:59.847681 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:05:59.847710 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:05:59.848121 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:59.848138 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.850313 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.850319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:59.853319 (+ 3000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:05:59.853332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.856077 (+ 2745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.856088 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:05:59.856356 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:59.856387 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 03:05:59.856482 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 03:05:59.856608 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12924,"mutex_wait_us":112,"thread_start_us":11488,"threads_started":1}
I20260620 03:05:59.857086 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:59.832644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.0.97.2:44573)
0620 03:05:59.847687 (+ 15043us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:59.847696 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:59.847721 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:05:59.847815 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:59.847822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:59.848038 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:59.848231 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:59.849063 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.850066 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.853041 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.854783 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.854903 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:59.855089 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:59.856581 (+ 1492us) server_negotiation.cc:299] Negotiation successful
0620 03:05:59.856753 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14854,"thread_start_us":15608,"threads_started":1}
I20260620 03:05:59.857765 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:05:59.833227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.0.97.3:46775)
0620 03:05:59.847048 (+ 13821us) server_negotiation.cc:206] Beginning negotiation
0620 03:05:59.847056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:05:59.847088 (+ 32us) server_negotiation.cc:407] Connection header received
0620 03:05:59.847195 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:05:59.847203 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:05:59.847457 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:05:59.848335 (+ 878us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:05:59.848597 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.850175 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.853656 (+ 3481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:05:59.855744 (+ 2088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:05:59.855902 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:05:59.856115 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 03:05:59.857288 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0620 03:05:59.857431 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13646,"thread_start_us":13492,"threads_started":1}
I20260620 03:05:59.860455 4641 tablet_service.cc:1467] Processing CreateTablet for tablet c6a63a220e8c46dda7e46f63f9c3540d (DEFAULT_TABLE table=test-table [id=47b8ebe079e94acbafee8ad67520993a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:59.860682 4774 tablet_service.cc:1467] Processing CreateTablet for tablet c6a63a220e8c46dda7e46f63f9c3540d (DEFAULT_TABLE table=test-table [id=47b8ebe079e94acbafee8ad67520993a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:59.861675 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a63a220e8c46dda7e46f63f9c3540d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:59.861964 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a63a220e8c46dda7e46f63f9c3540d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:59.861708 4906 tablet_service.cc:1467] Processing CreateTablet for tablet c6a63a220e8c46dda7e46f63f9c3540d (DEFAULT_TABLE table=test-table [id=47b8ebe079e94acbafee8ad67520993a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:05:59.862913 4906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a63a220e8c46dda7e46f63f9c3540d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:05:59.873242 4989 tablet_bootstrap.cc:492] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a: Bootstrap starting.
I20260620 03:05:59.876618 4989 tablet_bootstrap.cc:654] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:59.877502 4990 tablet_bootstrap.cc:492] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c: Bootstrap starting.
I20260620 03:05:59.877983 4989 log.cc:824] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:59.878372 4991 tablet_bootstrap.cc:492] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38: Bootstrap starting.
I20260620 03:05:59.880842 4989 tablet_bootstrap.cc:492] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a: No bootstrap required, opened a new log
I20260620 03:05:59.881126 4989 ts_tablet_manager.cc:1397] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 03:05:59.882110 4990 tablet_bootstrap.cc:654] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:59.882131 4991 tablet_bootstrap.cc:654] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38: Neither blocks nor log segments found. Creating new log.
I20260620 03:05:59.883349 4991 log.cc:824] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:59.883375 4990 log.cc:824] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c: Log is configured to *not* fsync() on all Append() calls
I20260620 03:05:59.889722 4990 tablet_bootstrap.cc:492] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c: No bootstrap required, opened a new log
I20260620 03:05:59.889520 4989 raft_consensus.cc:348] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c900c6658274e76be8d786920ee084c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44573 } } peers { permanent_uuid: "1927d27f3e2f4698a50a4f7545328f38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46775 } } peers { permanent_uuid: "8cde57d548384c2585da6d7fe9342a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46615 } }
I20260620 03:05:59.889954 4989 raft_consensus.cc:374] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:59.890058 4990 ts_tablet_manager.cc:1397] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260620 03:05:59.890057 4989 raft_consensus.cc:729] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cde57d548384c2585da6d7fe9342a4a, State: Initialized, Role: FOLLOWER
I20260620 03:05:59.890166 4991 tablet_bootstrap.cc:492] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38: No bootstrap required, opened a new log
I20260620 03:05:59.890416 4991 ts_tablet_manager.cc:1397] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260620 03:05:59.890715 4989 consensus_queue.cc:260] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c900c6658274e76be8d786920ee084c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44573 } } peers { permanent_uuid: "1927d27f3e2f4698a50a4f7545328f38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46775 } } peers { permanent_uuid: "8cde57d548384c2585da6d7fe9342a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46615 } }
I20260620 03:05:59.892554 4989 ts_tablet_manager.cc:1428] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a: Time spent starting tablet: real 0.011s user 0.008s sys 0.005s
I20260620 03:05:59.899384 4990 raft_consensus.cc:348] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c900c6658274e76be8d786920ee084c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44573 } } peers { permanent_uuid: "1927d27f3e2f4698a50a4f7545328f38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46775 } } peers { permanent_uuid: "8cde57d548384c2585da6d7fe9342a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46615 } }
I20260620 03:05:59.899767 4990 raft_consensus.cc:374] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:59.899878 4990 raft_consensus.cc:729] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c900c6658274e76be8d786920ee084c, State: Initialized, Role: FOLLOWER
I20260620 03:05:59.899647 4991 raft_consensus.cc:348] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c900c6658274e76be8d786920ee084c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44573 } } peers { permanent_uuid: "1927d27f3e2f4698a50a4f7545328f38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46775 } } peers { permanent_uuid: "8cde57d548384c2585da6d7fe9342a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46615 } }
I20260620 03:05:59.900056 4991 raft_consensus.cc:374] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:05:59.900174 4991 raft_consensus.cc:729] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1927d27f3e2f4698a50a4f7545328f38, State: Initialized, Role: FOLLOWER
I20260620 03:05:59.900460 4990 consensus_queue.cc:260] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c900c6658274e76be8d786920ee084c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44573 } } peers { permanent_uuid: "1927d27f3e2f4698a50a4f7545328f38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46775 } } peers { permanent_uuid: "8cde57d548384c2585da6d7fe9342a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46615 } }
I20260620 03:05:59.900704 4991 consensus_queue.cc:260] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c900c6658274e76be8d786920ee084c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44573 } } peers { permanent_uuid: "1927d27f3e2f4698a50a4f7545328f38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46775 } } peers { permanent_uuid: "8cde57d548384c2585da6d7fe9342a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46615 } }
I20260620 03:05:59.901540 4839 heartbeater.cc:503] Master 127.0.97.62:34621 was elected leader, sending a full tablet report...
I20260620 03:05:59.902117 4991 ts_tablet_manager.cc:1428] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260620 03:05:59.902172 4990 ts_tablet_manager.cc:1428] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 03:05:59.902505 4971 heartbeater.cc:503] Master 127.0.97.62:34621 was elected leader, sending a full tablet report...
W20260620 03:06:00.007498 4972 tablet.cc:2369] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:06:00.013525 4707 tablet.cc:2369] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:06:00.018098 4840 tablet.cc:2369] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:06:00.093829 4997 raft_consensus.cc:482] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:06:00.094161 4997 raft_consensus.cc:504] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c900c6658274e76be8d786920ee084c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44573 } } peers { permanent_uuid: "1927d27f3e2f4698a50a4f7545328f38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46775 } } peers { permanent_uuid: "8cde57d548384c2585da6d7fe9342a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46615 } }
I20260620 03:06:00.095933 4997 leader_election.cc:288] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c900c6658274e76be8d786920ee084c (127.0.97.2:44573), 8cde57d548384c2585da6d7fe9342a4a (127.0.97.1:46615)
I20260620 03:06:00.102494 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:00.096372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53711 (local address 127.0.97.2:44573)
0620 03:06:00.096548 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:00.096553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:00.096591 (+ 38us) server_negotiation.cc:407] Connection header received
0620 03:06:00.096765 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:00.096770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:00.096846 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:00.096955 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:00.097473 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:00.098551 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:00.100497 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:00.101662 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:00.101792 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:00.101885 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:00.102160 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 03:06:00.102312 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:06:00.102573 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:00.096248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44573 (local address 127.0.97.3:53711)
0620 03:06:00.096466 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:00.096488 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:00.096613 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:00.097148 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:00.097151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:00.097178 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:00.097324 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:00.097331 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:00.098693 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:00.098699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:00.100342 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:00.100354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:00.101809 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:00.101815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:00.102002 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:00.102038 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 03:06:00.102153 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 03:06:00.102286 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 03:06:00.103621 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:00.096814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46823 (local address 127.0.97.1:46615)
0620 03:06:00.096996 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:00.097004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:00.097484 (+ 480us) server_negotiation.cc:407] Connection header received
0620 03:06:00.097631 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:00.097635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:00.097690 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:00.097759 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:00.098369 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:00.099443 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:00.101495 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:00.102592 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:00.102757 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:00.102867 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:00.103246 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 03:06:00.103388 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 03:06:00.103720 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:00.096660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46615 (local address 127.0.97.3:46823)
0620 03:06:00.097349 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:00.097363 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:00.097498 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:00.097810 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:00.097815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:00.097840 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:00.098177 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:00.098187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:00.099582 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:00.099585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:00.101340 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:00.101354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:00.102884 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:00.102889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:00.103070 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:00.103102 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 03:06:00.103217 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 03:06:00.103356 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":3,"thread_start_us":352,"threads_started":1}
I20260620 03:06:00.103610 4794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6a63a220e8c46dda7e46f63f9c3540d" candidate_uuid: "1927d27f3e2f4698a50a4f7545328f38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c900c6658274e76be8d786920ee084c" is_pre_election: true
I20260620 03:06:00.104180 4794 raft_consensus.cc:2454] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1927d27f3e2f4698a50a4f7545328f38 in term 0.
I20260620 03:06:00.104631 4661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6a63a220e8c46dda7e46f63f9c3540d" candidate_uuid: "1927d27f3e2f4698a50a4f7545328f38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cde57d548384c2585da6d7fe9342a4a" is_pre_election: true
I20260620 03:06:00.104851 4861 leader_election.cc:302] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1927d27f3e2f4698a50a4f7545328f38, 9c900c6658274e76be8d786920ee084c; no voters:
I20260620 03:06:00.105135 4661 raft_consensus.cc:2454] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1927d27f3e2f4698a50a4f7545328f38 in term 0.
I20260620 03:06:00.105269 4997 raft_consensus.cc:2781] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:06:00.105419 4997 raft_consensus.cc:482] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:06:00.105530 4997 raft_consensus.cc:3037] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:00.107740 4997 raft_consensus.cc:504] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c900c6658274e76be8d786920ee084c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44573 } } peers { permanent_uuid: "1927d27f3e2f4698a50a4f7545328f38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46775 } } peers { permanent_uuid: "8cde57d548384c2585da6d7fe9342a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46615 } }
I20260620 03:06:00.108357 4997 leader_election.cc:288] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [CANDIDATE]: Term 1 election: Requested vote from peers 9c900c6658274e76be8d786920ee084c (127.0.97.2:44573), 8cde57d548384c2585da6d7fe9342a4a (127.0.97.1:46615)
I20260620 03:06:00.108680 4794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6a63a220e8c46dda7e46f63f9c3540d" candidate_uuid: "1927d27f3e2f4698a50a4f7545328f38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c900c6658274e76be8d786920ee084c"
I20260620 03:06:00.108747 4661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6a63a220e8c46dda7e46f63f9c3540d" candidate_uuid: "1927d27f3e2f4698a50a4f7545328f38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cde57d548384c2585da6d7fe9342a4a"
I20260620 03:06:00.108887 4794 raft_consensus.cc:3037] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:00.108963 4661 raft_consensus.cc:3037] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:00.111011 4794 raft_consensus.cc:2454] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1927d27f3e2f4698a50a4f7545328f38 in term 1.
I20260620 03:06:00.111104 4661 raft_consensus.cc:2454] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1927d27f3e2f4698a50a4f7545328f38 in term 1.
I20260620 03:06:00.111462 4861 leader_election.cc:302] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1927d27f3e2f4698a50a4f7545328f38, 9c900c6658274e76be8d786920ee084c; no voters:
I20260620 03:06:00.111755 4997 raft_consensus.cc:2781] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:00.112180 4997 raft_consensus.cc:686] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [term 1 LEADER]: Becoming Leader. State: Replica: 1927d27f3e2f4698a50a4f7545328f38, State: Running, Role: LEADER
I20260620 03:06:00.112637 4997 consensus_queue.cc:237] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c900c6658274e76be8d786920ee084c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44573 } } peers { permanent_uuid: "1927d27f3e2f4698a50a4f7545328f38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46775 } } peers { permanent_uuid: "8cde57d548384c2585da6d7fe9342a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46615 } }
I20260620 03:06:00.117223 4517 catalog_manager.cc:5495] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1927d27f3e2f4698a50a4f7545328f38 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1927d27f3e2f4698a50a4f7545328f38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c900c6658274e76be8d786920ee084c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1927d27f3e2f4698a50a4f7545328f38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cde57d548384c2585da6d7fe9342a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46615 } health_report { overall_health: UNKNOWN } } }
I20260620 03:06:00.150333 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:00.145120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.0.97.3:46775)
0620 03:06:00.145362 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:00.145368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:00.145497 (+ 129us) server_negotiation.cc:407] Connection header received
0620 03:06:00.145678 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:00.145682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:00.145756 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:00.145878 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:06:00.146187 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:00.146893 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:00.148288 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:00.148690 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:00.148800 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:00.149265 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:06:00.149768 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:06:00.149894 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:00.149954 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 03:06:00.150028 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 03:06:00.158501 4794 raft_consensus.cc:1261] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c [term 1 FOLLOWER]: Refusing update from remote peer 1927d27f3e2f4698a50a4f7545328f38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:06:00.158504 4661 raft_consensus.cc:1261] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a [term 1 FOLLOWER]: Refusing update from remote peer 1927d27f3e2f4698a50a4f7545328f38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:06:00.159463 4997 consensus_queue.cc:1035] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c900c6658274e76be8d786920ee084c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:00.159788 4999 consensus_queue.cc:1035] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cde57d548384c2585da6d7fe9342a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:00.188220 4518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:06:00.188545 4518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:06:00.191975 4518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4851aa02718e4699ad07f575d3049b07: Sending DeleteTablet for 3 replicas of tablet c6a63a220e8c46dda7e46f63f9c3540d
I20260620 03:06:00.193199 4774 tablet_service.cc:1514] Processing DeleteTablet for tablet c6a63a220e8c46dda7e46f63f9c3540d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50920
I20260620 03:06:00.193629 4641 tablet_service.cc:1514] Processing DeleteTablet for tablet c6a63a220e8c46dda7e46f63f9c3540d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45500
I20260620 03:06:00.194770 4906 tablet_service.cc:1514] Processing DeleteTablet for tablet c6a63a220e8c46dda7e46f63f9c3540d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60662
I20260620 03:06:00.199813 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 4579
I20260620 03:06:00.204802 5014 tablet_replica.cc:330] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38: stopping tablet replica
I20260620 03:06:00.205462 5014 raft_consensus.cc:2229] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:06:00.206401 5014 raft_consensus.cc:2258] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:00.208106 5013 tablet_replica.cc:330] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a: stopping tablet replica
I20260620 03:06:00.208508 5013 raft_consensus.cc:2229] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:06:00.208984 5013 raft_consensus.cc:2258] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:00.209240 5014 ts_tablet_manager.cc:1905] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:00.211985 5013 ts_tablet_manager.cc:1905] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:00.216902 5012 tablet_replica.cc:330] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c: stopping tablet replica
I20260620 03:06:00.217427 5012 raft_consensus.cc:2229] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:06:00.218040 5012 raft_consensus.cc:2258] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:00.219044 5014 ts_tablet_manager.cc:1918] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:06:00.219281 5014 log.cc:1196] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/c6a63a220e8c46dda7e46f63f9c3540d
I20260620 03:06:00.219998 5014 ts_tablet_manager.cc:1939] T c6a63a220e8c46dda7e46f63f9c3540d P 1927d27f3e2f4698a50a4f7545328f38: Deleting consensus metadata
I20260620 03:06:00.221030 5013 ts_tablet_manager.cc:1918] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:06:00.221217 5013 log.cc:1196] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/c6a63a220e8c46dda7e46f63f9c3540d
I20260620 03:06:00.221756 5013 ts_tablet_manager.cc:1939] T c6a63a220e8c46dda7e46f63f9c3540d P 8cde57d548384c2585da6d7fe9342a4a: Deleting consensus metadata
I20260620 03:06:00.222172 4502 catalog_manager.cc:4841] TS 1927d27f3e2f4698a50a4f7545328f38 (127.0.97.3:46775): tablet c6a63a220e8c46dda7e46f63f9c3540d (table test-table [id=47b8ebe079e94acbafee8ad67520993a]) successfully deleted
I20260620 03:06:00.223526 4504 catalog_manager.cc:4841] TS 8cde57d548384c2585da6d7fe9342a4a (127.0.97.1:46615): tablet c6a63a220e8c46dda7e46f63f9c3540d (table test-table [id=47b8ebe079e94acbafee8ad67520993a]) successfully deleted
I20260620 03:06:00.225380 5012 ts_tablet_manager.cc:1905] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 03:06:00 dist-test-slave-xv68 krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:06:00.232405 5012 ts_tablet_manager.cc:1918] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:06:00.232582 5012 log.cc:1196] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/c6a63a220e8c46dda7e46f63f9c3540d
I20260620 03:06:00.233088 5012 ts_tablet_manager.cc:1939] T c6a63a220e8c46dda7e46f63f9c3540d P 9c900c6658274e76be8d786920ee084c: Deleting consensus metadata
I20260620 03:06:00.234956 4504 catalog_manager.cc:4841] TS 9c900c6658274e76be8d786920ee084c (127.0.97.2:44573): tablet c6a63a220e8c46dda7e46f63f9c3540d (table test-table [id=47b8ebe079e94acbafee8ad67520993a]) successfully deleted
I20260620 03:06:00.235543 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:00.201222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45506 (local address 127.0.97.1:46615)
0620 03:06:00.201454 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:00.201463 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:00.219066 (+ 17603us) server_negotiation.cc:407] Connection header received
0620 03:06:00.219673 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:00.219681 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:00.219795 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:00.219879 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:00.221698 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:00.222756 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:00.224445 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:00.224962 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:00.225100 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:00.229103 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:00.229116 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:00.229135 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:00.229180 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:00.232116 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:00.232918 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:00.232925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:00.232928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:00.233061 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:00.233504 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:00.233509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:00.233512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:00.233855 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:00.234163 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:00.234452 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 03:06:00.235199 (+ 747us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 03:06:00.236199 4701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:00.325275 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 4579
I20260620 03:06:00.348544 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 4712
Jun 20 03:06:00 dist-test-slave-xv68 krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:06:00.363651 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:00.349481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50930 (local address 127.0.97.2:44573)
0620 03:06:00.349673 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:00.349689 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:00.349709 (+ 20us) server_negotiation.cc:407] Connection header received
0620 03:06:00.349775 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:00.349778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:00.349845 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:00.349952 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:00.351117 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:00.351937 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:00.353126 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:00.353515 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:00.353645 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:00.358102 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:00.358115 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:00.358130 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:00.358174 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:00.360927 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:00.361585 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:00.361593 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:00.361595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:00.361725 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:00.362071 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:00.362077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:00.362080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:00.362567 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:00.362814 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:00.363169 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 03:06:00.363394 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 03:06:00.364455 4834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:00.453117 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 4712
I20260620 03:06:00.470552 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 4844
Jun 20 03:06:00 dist-test-slave-xv68 krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:06:00.487706 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:00.471318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.0.97.3:46775)
0620 03:06:00.471524 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:00.471530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:00.471553 (+ 23us) server_negotiation.cc:407] Connection header received
0620 03:06:00.471692 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:00.471697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:00.471796 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:00.471925 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:00.473111 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:00.474366 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:00.475760 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:00.476425 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:00.476549 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:00.481163 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:00.481177 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:00.481193 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:00.481230 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:00.484562 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:00.485428 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:00.485436 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:00.485442 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:00.485592 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:00.486049 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:00.486053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:00.486055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:00.486425 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:00.486645 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:00.487039 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 03:06:00.487234 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 03:06:00.488441 4966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:00.607488 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 4844
I20260620 03:06:00.626372 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 4487
I20260620 03:06:00.627627 4548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:00.734922 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 4487
2026-06-20T03:06:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4246 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T03:06:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:06:00Z Disabled control of system clock
I20260620 03:06:00.801426 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43851
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 03:06:01.121528 5023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:01.121807 5023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:01.121891 5023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:01.129051 5023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:06:01.129160 5023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:01.129197 5023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:01.129252 5023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:06:01.129297 5023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:06:01.139494 5023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43851
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:01.140728 5023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:01.142400 5023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:01.153568 5028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:01.153541 5031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:01.153645 5029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:01.154772 5023 server_base.cc:1034] running on GCE node
I20260620 03:06:01.155535 5023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:01.156751 5023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:01.157990 5023 hybrid_clock.cc:648] HybridClock initialized: now 1781924761157917 us; error 56 us; skew 500 ppm
I20260620 03:06:01.158330 5023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:06:01.160555 5023 webserver.cc:460] Webserver started at http://127.0.97.62:45387/ using document root <none> and password file <none>
I20260620 03:06:01.161154 5023 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:01.161247 5023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:01.161548 5023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:01.164330 5023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "f6ea4a197750414795f10f93c9ac05b8"
format_stamp: "Formatted at 2026-06-20 03:06:01 on dist-test-slave-xv68"
server_key: "e959abeab5499b90b7de7196a5f5cb63"
server_key_iv: "0eda7892bc5eb48b1b9e016989fe4553"
server_key_version: "encryptionkey@0"
I20260620 03:06:01.165030 5023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "f6ea4a197750414795f10f93c9ac05b8"
format_stamp: "Formatted at 2026-06-20 03:06:01 on dist-test-slave-xv68"
server_key: "e959abeab5499b90b7de7196a5f5cb63"
server_key_iv: "0eda7892bc5eb48b1b9e016989fe4553"
server_key_version: "encryptionkey@0"
I20260620 03:06:01.170029 5023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 03:06:01.173978 5037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:01.175422 5023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:06:01.175583 5023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "f6ea4a197750414795f10f93c9ac05b8"
format_stamp: "Formatted at 2026-06-20 03:06:01 on dist-test-slave-xv68"
server_key: "e959abeab5499b90b7de7196a5f5cb63"
server_key_iv: "0eda7892bc5eb48b1b9e016989fe4553"
server_key_version: "encryptionkey@0"
I20260620 03:06:01.175806 5023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:01.181937 5023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:01.182745 5023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:01.183121 5023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:01.201912 5023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43851
I20260620 03:06:01.201946 5088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43851 every 8 connection(s)
I20260620 03:06:01.203346 5023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:06:01.207016 5089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:01.209975 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 5023
I20260620 03:06:01.210206 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:06:01.210990 388 external_mini_cluster.cc:1428] Setting key c37381c09f63b1ba9df45bbc8fdfe149
I20260620 03:06:01.219561 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:01.213232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59750 (local address 127.0.97.62:43851)
0620 03:06:01.214208 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:01.214219 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:01.214261 (+ 42us) server_negotiation.cc:407] Connection header received
0620 03:06:01.215955 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:01.215977 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:01.216424 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:01.216831 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:01.217896 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:01.217902 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:01.217914 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:01.217927 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:01.217980 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:01.217995 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:01.218222 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:01.218357 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 03:06:01.218601 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":350,"threads_started":1}
I20260620 03:06:01.219566 5089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8: Bootstrap starting.
I20260620 03:06:01.222994 5089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:01.224107 5089 log.cc:824] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:01.226871 5089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8: No bootstrap required, opened a new log
I20260620 03:06:01.234019 5089 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ea4a197750414795f10f93c9ac05b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43851 } }
I20260620 03:06:01.234404 5089 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:01.234510 5089 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ea4a197750414795f10f93c9ac05b8, State: Initialized, Role: FOLLOWER
I20260620 03:06:01.235100 5089 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ea4a197750414795f10f93c9ac05b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43851 } }
I20260620 03:06:01.235357 5089 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:06:01.235493 5089 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:06:01.235663 5089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:01.238010 5089 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ea4a197750414795f10f93c9ac05b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43851 } }
I20260620 03:06:01.238493 5089 leader_election.cc:302] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6ea4a197750414795f10f93c9ac05b8; no voters:
I20260620 03:06:01.239040 5089 leader_election.cc:288] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:06:01.239203 5094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:01.239964 5094 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8 [term 1 LEADER]: Becoming Leader. State: Replica: f6ea4a197750414795f10f93c9ac05b8, State: Running, Role: LEADER
I20260620 03:06:01.240589 5094 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ea4a197750414795f10f93c9ac05b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43851 } }
I20260620 03:06:01.241094 5089 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:06:01.243551 5095 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6ea4a197750414795f10f93c9ac05b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ea4a197750414795f10f93c9ac05b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43851 } } }
I20260620 03:06:01.243654 5096 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6ea4a197750414795f10f93c9ac05b8. Latest consensus state: current_term: 1 leader_uuid: "f6ea4a197750414795f10f93c9ac05b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ea4a197750414795f10f93c9ac05b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43851 } } }
I20260620 03:06:01.243960 5095 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:01.244031 5096 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:01.246851 5100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:06:01.251564 5100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:06:01.259649 5100 catalog_manager.cc:1347] Generated new cluster ID: 0e538fef257d4ebc87c030848c0b0f2b
I20260620 03:06:01.259773 5100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:06:01.275918 5100 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:06:01.276571 5100 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:06:01.290627 5100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8: Generated new TSK 0
I20260620 03:06:01.291563 5100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:06:01.303253 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43851
--builtin_ntp_servers=127.0.97.20:41819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 03:06:01.639905 5113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:01.640259 5113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:01.640419 5113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:01.647647 5113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:01.647759 5113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:01.647943 5113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:06:01.659458 5113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:01.661340 5113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:01.663247 5113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:01.677472 5118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:01.677338 5119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:01.677445 5121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:01.678175 5113 server_base.cc:1034] running on GCE node
I20260620 03:06:01.678720 5113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:01.679720 5113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:01.680917 5113 hybrid_clock.cc:648] HybridClock initialized: now 1781924761680857 us; error 47 us; skew 500 ppm
I20260620 03:06:01.681217 5113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:06:01.683514 5113 webserver.cc:460] Webserver started at http://127.0.97.1:42105/ using document root <none> and password file <none>
I20260620 03:06:01.684166 5113 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:01.684263 5113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:01.684514 5113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:01.687537 5113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "4a2b0edf349e4790b0814243768e3932"
format_stamp: "Formatted at 2026-06-20 03:06:01 on dist-test-slave-xv68"
server_key: "babce909cc78243f55bf332238ccf930"
server_key_iv: "f92d6bbfe4c8fa4fa958acdb41c10bd0"
server_key_version: "encryptionkey@0"
I20260620 03:06:01.688297 5113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4a2b0edf349e4790b0814243768e3932"
format_stamp: "Formatted at 2026-06-20 03:06:01 on dist-test-slave-xv68"
server_key: "babce909cc78243f55bf332238ccf930"
server_key_iv: "f92d6bbfe4c8fa4fa958acdb41c10bd0"
server_key_version: "encryptionkey@0"
I20260620 03:06:01.693197 5113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:06:01.696534 5127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:01.697892 5113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:06:01.698079 5113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "4a2b0edf349e4790b0814243768e3932"
format_stamp: "Formatted at 2026-06-20 03:06:01 on dist-test-slave-xv68"
server_key: "babce909cc78243f55bf332238ccf930"
server_key_iv: "f92d6bbfe4c8fa4fa958acdb41c10bd0"
server_key_version: "encryptionkey@0"
I20260620 03:06:01.698318 5113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:01.704037 5113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:01.704789 5113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:01.705056 5113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:01.705968 5113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:01.707422 5113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:01.707536 5113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:01.707669 5113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:01.707767 5113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:01.741142 5113 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38947
I20260620 03:06:01.741252 5239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38947 every 8 connection(s)
I20260620 03:06:01.742551 5113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:06:01.750706 5242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:01.745154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45481 (local address 127.0.97.62:43851)
0620 03:06:01.745652 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:01.745658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:01.746735 (+ 1077us) server_negotiation.cc:407] Connection header received
0620 03:06:01.748454 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:01.748459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:01.748531 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:01.748645 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:01.749957 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:01.749963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:01.749968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:01.749985 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:01.750033 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:01.750044 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:01.750170 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:01.750359 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 03:06:01.750421 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":259,"threads_started":1}
I20260620 03:06:01.751632 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:01.745652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43851 (local address 127.0.97.1:45481)
0620 03:06:01.746579 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:01.746621 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:01.748205 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:01.748894 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:01.748902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:01.748990 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:01.749671 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:06:01.749673 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:01.749711 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:06:01.749792 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:01.750142 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:01.750153 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:01.750160 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 03:06:01.750439 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 03:06:01.750676 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":241,"threads_started":1}
I20260620 03:06:01.752518 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 5113
I20260620 03:06:01.752702 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:06:01.753464 388 external_mini_cluster.cc:1428] Setting key 9096c323e6520e157f95190812e6d31a
I20260620 03:06:01.753515 5240 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43851
I20260620 03:06:01.753826 5240 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:01.754462 5240 heartbeater.cc:511] Master 127.0.97.62:43851 requested a full tablet report, sending...
I20260620 03:06:01.756415 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43851
--builtin_ntp_servers=127.0.97.20:41819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 03:06:01.756675 5054 ts_manager.cc:194] Registered new tserver with Master: 4a2b0edf349e4790b0814243768e3932 (127.0.97.1:38947)
W20260620 03:06:02.070892 5244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:02.071202 5244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:02.071398 5244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:02.078341 5244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:02.078490 5244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:02.078672 5244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:06:02.088069 5244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:02.089351 5244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:02.091058 5244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:02.104168 5249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:02.104190 5252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:02.104125 5250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:02.104912 5244 server_base.cc:1034] running on GCE node
I20260620 03:06:02.105455 5244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:02.106314 5244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:02.107506 5244 hybrid_clock.cc:648] HybridClock initialized: now 1781924762107464 us; error 35 us; skew 500 ppm
I20260620 03:06:02.107813 5244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:06:02.109997 5244 webserver.cc:460] Webserver started at http://127.0.97.2:43785/ using document root <none> and password file <none>
I20260620 03:06:02.110647 5244 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:02.110729 5244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:02.111070 5244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:02.113857 5244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "4f53db8385574da0bd610b473bb148a0"
format_stamp: "Formatted at 2026-06-20 03:06:02 on dist-test-slave-xv68"
server_key: "aa83be1afd692f420491e3c5ed200599"
server_key_iv: "f9efcfea80bfe3babc20eb36e98e698b"
server_key_version: "encryptionkey@0"
I20260620 03:06:02.114619 5244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4f53db8385574da0bd610b473bb148a0"
format_stamp: "Formatted at 2026-06-20 03:06:02 on dist-test-slave-xv68"
server_key: "aa83be1afd692f420491e3c5ed200599"
server_key_iv: "f9efcfea80bfe3babc20eb36e98e698b"
server_key_version: "encryptionkey@0"
I20260620 03:06:02.119419 5244 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260620 03:06:02.122676 5258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:02.124032 5244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:06:02.124199 5244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "4f53db8385574da0bd610b473bb148a0"
format_stamp: "Formatted at 2026-06-20 03:06:02 on dist-test-slave-xv68"
server_key: "aa83be1afd692f420491e3c5ed200599"
server_key_iv: "f9efcfea80bfe3babc20eb36e98e698b"
server_key_version: "encryptionkey@0"
I20260620 03:06:02.124428 5244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:02.129946 5244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:02.130722 5244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:02.130986 5244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:02.131942 5244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:02.133353 5244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:02.133474 5244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:02.133603 5244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:02.133689 5244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:02.166396 5244 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36005
I20260620 03:06:02.166483 5370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36005 every 8 connection(s)
I20260620 03:06:02.167821 5244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:06:02.174932 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 5244
I20260620 03:06:02.175113 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:06:02.175731 388 external_mini_cluster.cc:1428] Setting key 80a99430d74305682ebbc9efc70a2fb3
I20260620 03:06:02.176124 5242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.170415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51071 (local address 127.0.97.62:43851)
0620 03:06:02.170617 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:02.170623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:02.172132 (+ 1509us) server_negotiation.cc:407] Connection header received
0620 03:06:02.173830 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:02.173834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:02.173881 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:02.173968 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:02.175383 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:02.175387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:02.175390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:02.175404 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:02.175432 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:02.175443 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:02.175585 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:02.175781 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 03:06:02.175844 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 03:06:02.177220 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.170987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43851 (local address 127.0.97.2:51071)
0620 03:06:02.171979 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:02.172026 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:02.173585 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:02.174176 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:02.174184 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:02.174279 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:02.174979 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:06:02.174982 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:02.175039 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:06:02.175166 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:02.175561 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:02.175572 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:02.175580 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 03:06:02.175884 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 03:06:02.176140 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":343,"threads_started":1}
I20260620 03:06:02.178632 5371 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43851
I20260620 03:06:02.178753 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43851
--builtin_ntp_servers=127.0.97.20:41819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 03:06:02.179000 5371 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:02.179790 5371 heartbeater.cc:511] Master 127.0.97.62:43851 requested a full tablet report, sending...
I20260620 03:06:02.181596 5054 ts_manager.cc:194] Registered new tserver with Master: 4f53db8385574da0bd610b473bb148a0 (127.0.97.2:36005)
W20260620 03:06:02.500641 5374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:02.500967 5374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:02.501125 5374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:02.508457 5374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:02.508584 5374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:02.508766 5374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:06:02.517766 5374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:02.519029 5374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:02.520817 5374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:02.533615 5379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:02.533525 5380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:02.533649 5382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:02.534277 5374 server_base.cc:1034] running on GCE node
I20260620 03:06:02.534901 5374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:02.535878 5374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:02.537108 5374 hybrid_clock.cc:648] HybridClock initialized: now 1781924762537071 us; error 51 us; skew 500 ppm
I20260620 03:06:02.537415 5374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:06:02.539710 5374 webserver.cc:460] Webserver started at http://127.0.97.3:34385/ using document root <none> and password file <none>
I20260620 03:06:02.540377 5374 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:02.540504 5374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:02.540813 5374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:02.543906 5374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "b03c38008db0448595171573d2ba03b3"
format_stamp: "Formatted at 2026-06-20 03:06:02 on dist-test-slave-xv68"
server_key: "ab256564f559c28968a44407d405b7b1"
server_key_iv: "bd9f51d00ac49131c37969b568d3851b"
server_key_version: "encryptionkey@0"
I20260620 03:06:02.544687 5374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b03c38008db0448595171573d2ba03b3"
format_stamp: "Formatted at 2026-06-20 03:06:02 on dist-test-slave-xv68"
server_key: "ab256564f559c28968a44407d405b7b1"
server_key_iv: "bd9f51d00ac49131c37969b568d3851b"
server_key_version: "encryptionkey@0"
I20260620 03:06:02.549657 5374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:06:02.553321 5388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:02.554571 5374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 03:06:02.554729 5374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "b03c38008db0448595171573d2ba03b3"
format_stamp: "Formatted at 2026-06-20 03:06:02 on dist-test-slave-xv68"
server_key: "ab256564f559c28968a44407d405b7b1"
server_key_iv: "bd9f51d00ac49131c37969b568d3851b"
server_key_version: "encryptionkey@0"
I20260620 03:06:02.554952 5374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:02.560312 5374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:02.561053 5374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:02.561305 5374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:02.562134 5374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:02.563482 5374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:02.563594 5374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:02.563722 5374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:02.563807 5374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:02.595966 5374 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44267
I20260620 03:06:02.596022 5500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44267 every 8 connection(s)
I20260620 03:06:02.597301 5374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:06:02.599056 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 5374
I20260620 03:06:02.599231 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:06:02.599803 388 external_mini_cluster.cc:1428] Setting key 810f4f4edf73e8a3428e6e2dfe2f9d9b
I20260620 03:06:02.606252 5242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.600220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37257 (local address 127.0.97.62:43851)
0620 03:06:02.600407 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:02.600411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:02.602212 (+ 1801us) server_negotiation.cc:407] Connection header received
0620 03:06:02.604104 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:02.604109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:02.604172 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:02.604281 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:02.605541 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:02.605546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:02.605551 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:02.605568 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:02.605617 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:02.605629 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:02.605749 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:02.605935 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 03:06:02.605997 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 03:06:02.607116 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.600906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43851 (local address 127.0.97.3:37257)
0620 03:06:02.602075 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:02.602117 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:02.603847 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:02.604491 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:02.604499 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:02.604585 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:02.605263 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:06:02.605265 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:02.605300 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:06:02.605383 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:02.605718 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:02.605729 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:02.605738 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 03:06:02.606014 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 03:06:02.606250 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":309,"threads_started":1}
I20260620 03:06:02.608270 5501 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43851
I20260620 03:06:02.608526 5501 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:02.608987 5501 heartbeater.cc:511] Master 127.0.97.62:43851 requested a full tablet report, sending...
I20260620 03:06:02.610190 5054 ts_manager.cc:194] Registered new tserver with Master: b03c38008db0448595171573d2ba03b3 (127.0.97.3:44267)
I20260620 03:06:02.614871 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:06:02.621487 5242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.619148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59754 (local address 127.0.97.62:43851)
0620 03:06:02.619270 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:02.619274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:02.619824 (+ 550us) server_negotiation.cc:407] Connection header received
0620 03:06:02.619939 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:02.619943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:02.619991 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:02.620070 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:02.620943 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:02.620947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:02.620950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:02.620963 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:02.620991 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:02.621001 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:02.621095 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:02.621125 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 03:06:02.621164 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 03:06:02.625200 5054 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:06:02.631884 5054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:06:02.656124 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.643450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44267 (local address 127.0.0.1:41940)
0620 03:06:02.644323 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:02.644364 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:02.644487 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:02.652825 (+ 8338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:02.652833 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:02.652928 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:02.653796 (+ 868us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:06:02.653802 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:02.653833 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:06:02.653925 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:02.655347 (+ 1422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:02.655356 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:02.655365 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 03:06:02.655530 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 03:06:02.655612 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":11426,"threads_started":1}
I20260620 03:06:02.656260 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.643675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36005 (local address 127.0.0.1:47444)
0620 03:06:02.652693 (+ 9018us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:02.652712 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:02.652816 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:02.654400 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:02.654404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:02.654425 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:02.655028 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:06:02.655031 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:02.655048 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:06:02.655088 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:02.655624 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:02.655629 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:02.655633 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 03:06:02.655804 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 03:06:02.655854 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8865,"thread_start_us":498,"threads_started":1}
I20260620 03:06:02.656936 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.644011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41940 (local address 127.0.97.3:44267)
0620 03:06:02.652183 (+ 8172us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:02.652195 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:02.652236 (+ 41us) server_negotiation.cc:407] Connection header received
0620 03:06:02.652321 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:02.652332 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:02.652596 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:02.653363 (+ 767us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:02.654757 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:02.655068 (+ 311us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:02.655084 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:02.655103 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:02.655166 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:02.655195 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:02.656402 (+ 1207us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:02.656474 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 03:06:02.656589 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7977,"thread_start_us":7834,"threads_started":1}
I20260620 03:06:02.656955 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.643968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47444 (local address 127.0.97.2:36005)
0620 03:06:02.652907 (+ 8939us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:02.652914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:02.652941 (+ 27us) server_negotiation.cc:407] Connection header received
0620 03:06:02.653007 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:02.653016 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:02.653219 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:02.653364 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:02.655251 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:02.655261 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:02.655273 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:02.655321 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:02.655380 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:02.655402 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:02.655571 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:02.656633 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0620 03:06:02.656705 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8781,"thread_start_us":8655,"threads_started":1}
I20260620 03:06:02.658865 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.645283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39156 (local address 127.0.97.1:38947)
0620 03:06:02.645826 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:02.645837 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:02.655518 (+ 9681us) server_negotiation.cc:407] Connection header received
0620 03:06:02.655741 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:02.655752 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:02.655951 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:02.656070 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:02.657948 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:02.657956 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:02.657968 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:02.657988 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:02.658044 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:02.658065 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:02.658209 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:02.658481 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 03:06:02.658571 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":277,"threads_started":1}
I20260620 03:06:02.658878 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.644982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38947 (local address 127.0.0.1:39156)
0620 03:06:02.655417 (+ 10435us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:02.655433 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:02.655594 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:02.656989 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:02.656995 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:02.657014 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:02.657704 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:06:02.657707 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:02.657721 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:06:02.657759 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:02.658300 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:02.658308 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:02.658315 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 03:06:02.658470 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 03:06:02.658519 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10319,"thread_start_us":7400,"threads_started":1}
I20260620 03:06:02.661074 5306 tablet_service.cc:1467] Processing CreateTablet for tablet 3277b94ccd334a7a9139445397d32a7a (DEFAULT_TABLE table=test-table [id=2235fb6125b648dd85676b8a86983e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:02.661134 5436 tablet_service.cc:1467] Processing CreateTablet for tablet 3277b94ccd334a7a9139445397d32a7a (DEFAULT_TABLE table=test-table [id=2235fb6125b648dd85676b8a86983e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:02.662269 5306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3277b94ccd334a7a9139445397d32a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:02.662438 5436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3277b94ccd334a7a9139445397d32a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:02.661990 5175 tablet_service.cc:1467] Processing CreateTablet for tablet 3277b94ccd334a7a9139445397d32a7a (DEFAULT_TABLE table=test-table [id=2235fb6125b648dd85676b8a86983e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:02.663126 5175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3277b94ccd334a7a9139445397d32a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:02.673906 5519 tablet_bootstrap.cc:492] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0: Bootstrap starting.
I20260620 03:06:02.676141 5520 tablet_bootstrap.cc:492] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3: Bootstrap starting.
I20260620 03:06:02.677279 5519 tablet_bootstrap.cc:654] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:02.678592 5519 log.cc:824] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:02.679836 5520 tablet_bootstrap.cc:654] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:02.680761 5521 tablet_bootstrap.cc:492] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932: Bootstrap starting.
I20260620 03:06:02.681200 5520 log.cc:824] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:02.682641 5519 tablet_bootstrap.cc:492] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0: No bootstrap required, opened a new log
I20260620 03:06:02.682914 5519 ts_tablet_manager.cc:1397] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260620 03:06:02.684051 5521 tablet_bootstrap.cc:654] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:02.684262 5520 tablet_bootstrap.cc:492] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3: No bootstrap required, opened a new log
I20260620 03:06:02.684506 5520 ts_tablet_manager.cc:1397] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260620 03:06:02.685091 5521 log.cc:824] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:02.687934 5521 tablet_bootstrap.cc:492] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932: No bootstrap required, opened a new log
I20260620 03:06:02.688207 5521 ts_tablet_manager.cc:1397] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 03:06:02.693425 5519 raft_consensus.cc:348] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03c38008db0448595171573d2ba03b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "4f53db8385574da0bd610b473bb148a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36005 } } peers { permanent_uuid: "4a2b0edf349e4790b0814243768e3932" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38947 } }
I20260620 03:06:02.693877 5519 raft_consensus.cc:374] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:02.694018 5519 raft_consensus.cc:729] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f53db8385574da0bd610b473bb148a0, State: Initialized, Role: FOLLOWER
I20260620 03:06:02.694659 5519 consensus_queue.cc:260] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03c38008db0448595171573d2ba03b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "4f53db8385574da0bd610b473bb148a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36005 } } peers { permanent_uuid: "4a2b0edf349e4790b0814243768e3932" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38947 } }
I20260620 03:06:02.694753 5520 raft_consensus.cc:348] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03c38008db0448595171573d2ba03b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "4f53db8385574da0bd610b473bb148a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36005 } } peers { permanent_uuid: "4a2b0edf349e4790b0814243768e3932" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38947 } }
I20260620 03:06:02.695137 5520 raft_consensus.cc:374] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:02.695334 5520 raft_consensus.cc:729] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03c38008db0448595171573d2ba03b3, State: Initialized, Role: FOLLOWER
I20260620 03:06:02.695762 5371 heartbeater.cc:503] Master 127.0.97.62:43851 was elected leader, sending a full tablet report...
I20260620 03:06:02.695982 5521 raft_consensus.cc:348] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03c38008db0448595171573d2ba03b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "4f53db8385574da0bd610b473bb148a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36005 } } peers { permanent_uuid: "4a2b0edf349e4790b0814243768e3932" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38947 } }
I20260620 03:06:02.696070 5520 consensus_queue.cc:260] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03c38008db0448595171573d2ba03b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "4f53db8385574da0bd610b473bb148a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36005 } } peers { permanent_uuid: "4a2b0edf349e4790b0814243768e3932" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38947 } }
I20260620 03:06:02.696292 5521 raft_consensus.cc:374] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:02.696412 5521 raft_consensus.cc:729] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a2b0edf349e4790b0814243768e3932, State: Initialized, Role: FOLLOWER
I20260620 03:06:02.696929 5521 consensus_queue.cc:260] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03c38008db0448595171573d2ba03b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "4f53db8385574da0bd610b473bb148a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36005 } } peers { permanent_uuid: "4a2b0edf349e4790b0814243768e3932" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38947 } }
I20260620 03:06:02.698269 5519 ts_tablet_manager.cc:1428] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260620 03:06:02.699719 5501 heartbeater.cc:503] Master 127.0.97.62:43851 was elected leader, sending a full tablet report...
I20260620 03:06:02.699745 5240 heartbeater.cc:503] Master 127.0.97.62:43851 was elected leader, sending a full tablet report...
I20260620 03:06:02.700336 5521 ts_tablet_manager.cc:1428] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 03:06:02.701444 5520 ts_tablet_manager.cc:1428] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260620 03:06:02.722589 5527 raft_consensus.cc:482] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:06:02.722879 5527 raft_consensus.cc:504] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03c38008db0448595171573d2ba03b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "4f53db8385574da0bd610b473bb148a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36005 } } peers { permanent_uuid: "4a2b0edf349e4790b0814243768e3932" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38947 } }
I20260620 03:06:02.724547 5527 leader_election.cc:288] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03c38008db0448595171573d2ba03b3 (127.0.97.3:44267), 4f53db8385574da0bd610b473bb148a0 (127.0.97.2:36005)
I20260620 03:06:02.727936 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.725039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36471 (local address 127.0.97.3:44267)
0620 03:06:02.725365 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:02.725373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:02.725826 (+ 453us) server_negotiation.cc:407] Connection header received
0620 03:06:02.725984 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:02.725988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:02.726065 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:02.726167 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:02.727185 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:02.727193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:02.727202 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:02.727239 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:02.727278 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:02.727324 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:02.727462 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:02.727637 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 03:06:02.727695 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260620 03:06:02.728020 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.725056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55147 (local address 127.0.97.2:36005)
0620 03:06:02.725532 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:02.725540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:02.726034 (+ 494us) server_negotiation.cc:407] Connection header received
0620 03:06:02.726123 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:02.726131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:02.726183 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:02.726274 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:02.727240 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:02.727248 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:02.727254 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:02.727279 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:02.727356 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:02.727371 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:02.727510 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:02.727700 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 03:06:02.727757 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340}
I20260620 03:06:02.728286 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.724679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44267 (local address 127.0.97.1:36471)
0620 03:06:02.725676 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:02.725710 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:02.725867 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:02.726201 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:02.726204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:02.726221 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:02.726957 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:06:02.726961 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:02.726980 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:06:02.727037 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:02.727431 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:02.727439 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:02.727446 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 03:06:02.727648 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 03:06:02.727701 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":636,"threads_started":1}
I20260620 03:06:02.728647 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.724919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36005 (local address 127.0.97.1:55147)
0620 03:06:02.725940 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:02.725957 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:02.726043 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:02.726344 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:02.726354 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:02.726376 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:02.727051 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:06:02.727053 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:02.727090 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:06:02.727127 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:02.727560 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:02.727566 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:02.727572 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 03:06:02.728244 (+ 672us) client_negotiation.cc:240] Negotiation successful
0620 03:06:02.728302 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":685,"threads_started":1}
I20260620 03:06:02.729333 5456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3277b94ccd334a7a9139445397d32a7a" candidate_uuid: "4a2b0edf349e4790b0814243768e3932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03c38008db0448595171573d2ba03b3" is_pre_election: true
I20260620 03:06:02.729714 5326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3277b94ccd334a7a9139445397d32a7a" candidate_uuid: "4a2b0edf349e4790b0814243768e3932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f53db8385574da0bd610b473bb148a0" is_pre_election: true
I20260620 03:06:02.729827 5456 raft_consensus.cc:2454] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a2b0edf349e4790b0814243768e3932 in term 0.
I20260620 03:06:02.730132 5326 raft_consensus.cc:2454] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a2b0edf349e4790b0814243768e3932 in term 0.
I20260620 03:06:02.730451 5128 leader_election.cc:302] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2b0edf349e4790b0814243768e3932, b03c38008db0448595171573d2ba03b3; no voters:
I20260620 03:06:02.730834 5527 raft_consensus.cc:2781] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:06:02.730980 5527 raft_consensus.cc:482] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:06:02.731084 5527 raft_consensus.cc:3037] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:02.733201 5527 raft_consensus.cc:504] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03c38008db0448595171573d2ba03b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "4f53db8385574da0bd610b473bb148a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36005 } } peers { permanent_uuid: "4a2b0edf349e4790b0814243768e3932" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38947 } }
I20260620 03:06:02.733778 5527 leader_election.cc:288] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [CANDIDATE]: Term 1 election: Requested vote from peers b03c38008db0448595171573d2ba03b3 (127.0.97.3:44267), 4f53db8385574da0bd610b473bb148a0 (127.0.97.2:36005)
I20260620 03:06:02.734099 5456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3277b94ccd334a7a9139445397d32a7a" candidate_uuid: "4a2b0edf349e4790b0814243768e3932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03c38008db0448595171573d2ba03b3"
I20260620 03:06:02.734282 5456 raft_consensus.cc:3037] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:02.734269 5326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3277b94ccd334a7a9139445397d32a7a" candidate_uuid: "4a2b0edf349e4790b0814243768e3932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f53db8385574da0bd610b473bb148a0"
I20260620 03:06:02.734452 5326 raft_consensus.cc:3037] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:02.736550 5456 raft_consensus.cc:2454] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a2b0edf349e4790b0814243768e3932 in term 1.
I20260620 03:06:02.736570 5326 raft_consensus.cc:2454] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a2b0edf349e4790b0814243768e3932 in term 1.
I20260620 03:06:02.736889 5128 leader_election.cc:302] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2b0edf349e4790b0814243768e3932, b03c38008db0448595171573d2ba03b3; no voters:
I20260620 03:06:02.737146 5527 raft_consensus.cc:2781] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:02.737573 5527 raft_consensus.cc:686] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [term 1 LEADER]: Becoming Leader. State: Replica: 4a2b0edf349e4790b0814243768e3932, State: Running, Role: LEADER
I20260620 03:06:02.738019 5527 consensus_queue.cc:237] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03c38008db0448595171573d2ba03b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "4f53db8385574da0bd610b473bb148a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36005 } } peers { permanent_uuid: "4a2b0edf349e4790b0814243768e3932" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38947 } }
I20260620 03:06:02.742733 5053 catalog_manager.cc:5495] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a2b0edf349e4790b0814243768e3932 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "4a2b0edf349e4790b0814243768e3932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03c38008db0448595171573d2ba03b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f53db8385574da0bd610b473bb148a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2b0edf349e4790b0814243768e3932" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38947 } health_report { overall_health: HEALTHY } } }
W20260620 03:06:02.744731 5241 tablet.cc:2369] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:06:02.774480 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.772378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39172 (local address 127.0.97.1:38947)
0620 03:06:02.772552 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:02.772557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:02.772657 (+ 100us) server_negotiation.cc:407] Connection header received
0620 03:06:02.772824 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:02.772827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:02.772904 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:02.772999 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:02.773957 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:02.773961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:02.773965 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:02.773978 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:02.774007 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:02.774018 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:02.774106 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:02.774200 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 03:06:02.774243 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 03:06:02.782509 5326 raft_consensus.cc:1261] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0 [term 1 FOLLOWER]: Refusing update from remote peer 4a2b0edf349e4790b0814243768e3932: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:06:02.782510 5456 raft_consensus.cc:1261] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3 [term 1 FOLLOWER]: Refusing update from remote peer 4a2b0edf349e4790b0814243768e3932: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:06:02.783506 5530 consensus_queue.cc:1035] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53db8385574da0bd610b473bb148a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:02.783874 5527 consensus_queue.cc:1035] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03c38008db0448595171573d2ba03b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:02.795503 5533 mvcc.cc:205] Tried to move back new op lower bound from 7298763828347891712 to 7298763828178726912. Current Snapshot: MvccSnapshot[applied={T|T < 7298763828347891712}]
I20260620 03:06:02.808966 5052 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:06:02.809285 5052 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:06:02.812958 5052 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6ea4a197750414795f10f93c9ac05b8: Sending DeleteTablet for 3 replicas of tablet 3277b94ccd334a7a9139445397d32a7a
I20260620 03:06:02.814002 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet 3277b94ccd334a7a9139445397d32a7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:02 UTC) from {username='slave'} at 127.0.0.1:41940
I20260620 03:06:02.814217 5306 tablet_service.cc:1514] Processing DeleteTablet for tablet 3277b94ccd334a7a9139445397d32a7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:02 UTC) from {username='slave'} at 127.0.0.1:47444
I20260620 03:06:02.814460 5175 tablet_service.cc:1514] Processing DeleteTablet for tablet 3277b94ccd334a7a9139445397d32a7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:02 UTC) from {username='slave'} at 127.0.0.1:39156
I20260620 03:06:02.816787 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 5113
I20260620 03:06:02.826553 5543 tablet_replica.cc:330] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0: stopping tablet replica
I20260620 03:06:02.826676 5544 tablet_replica.cc:330] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932: stopping tablet replica
I20260620 03:06:02.827162 5543 raft_consensus.cc:2229] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:06:02.827338 5544 raft_consensus.cc:2229] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:06:02.827849 5543 raft_consensus.cc:2258] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:02.828181 5544 raft_consensus.cc:2258] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:02.829114 5542 tablet_replica.cc:330] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3: stopping tablet replica
I20260620 03:06:02.829559 5542 raft_consensus.cc:2229] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:06:02.830437 5542 raft_consensus.cc:2258] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:02.830971 5543 ts_tablet_manager.cc:1905] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:02.830973 5544 ts_tablet_manager.cc:1905] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:02.832712 5542 ts_tablet_manager.cc:1905] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:02.837026 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.819744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39188 (local address 127.0.97.1:38947)
0620 03:06:02.820312 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:02.820320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:02.828506 (+ 8186us) server_negotiation.cc:407] Connection header received
0620 03:06:02.829352 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:02.829357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:02.829417 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:02.829499 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:02.836143 (+ 6644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:02.836149 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:02.836154 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:02.836175 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:02.836218 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:02.836234 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:02.836342 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:02.836617 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 03:06:02.836684 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411}
I20260620 03:06:02.838333 5235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:02.839951 5542 ts_tablet_manager.cc:1918] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:06:02.840113 5542 log.cc:1196] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/3277b94ccd334a7a9139445397d32a7a
I20260620 03:06:02.840857 5542 ts_tablet_manager.cc:1939] T 3277b94ccd334a7a9139445397d32a7a P b03c38008db0448595171573d2ba03b3: Deleting consensus metadata
I20260620 03:06:02.841178 5543 ts_tablet_manager.cc:1918] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:06:02.841466 5543 log.cc:1196] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/3277b94ccd334a7a9139445397d32a7a
I20260620 03:06:02.842082 5543 ts_tablet_manager.cc:1939] T 3277b94ccd334a7a9139445397d32a7a P 4f53db8385574da0bd610b473bb148a0: Deleting consensus metadata
I20260620 03:06:02.843070 5038 catalog_manager.cc:4841] TS b03c38008db0448595171573d2ba03b3 (127.0.97.3:44267): tablet 3277b94ccd334a7a9139445397d32a7a (table test-table [id=2235fb6125b648dd85676b8a86983e6f]) successfully deleted
I20260620 03:06:02.844452 5040 catalog_manager.cc:4841] TS 4f53db8385574da0bd610b473bb148a0 (127.0.97.2:36005): tablet 3277b94ccd334a7a9139445397d32a7a (table test-table [id=2235fb6125b648dd85676b8a86983e6f]) successfully deleted
I20260620 03:06:02.939882 5544 ts_tablet_manager.cc:1918] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:06:02.940172 5544 log.cc:1196] T 3277b94ccd334a7a9139445397d32a7a P 4a2b0edf349e4790b0814243768e3932: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/3277b94ccd334a7a9139445397d32a7a
I20260620 03:06:02.940613 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 5113
I20260620 03:06:02.958099 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 5244
W20260620 03:06:02.958009 5040 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38947 (error 108)
I20260620 03:06:02.960475 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.959998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38947 (local address 127.0.0.1:39194)
0620 03:06:02.960181 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:02.960304 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 03:06:02.960790 5040 catalog_manager.cc:4584] TS 4a2b0edf349e4790b0814243768e3932 (127.0.97.1:38947): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3277b94ccd334a7a9139445397d32a7a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38947: connect: Connection refused (error 111)
I20260620 03:06:02.961130 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:02.958914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.0.97.2:36005)
0620 03:06:02.959109 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:02.959115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:02.959137 (+ 22us) server_negotiation.cc:407] Connection header received
0620 03:06:02.959378 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:02.959384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:02.959496 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:02.959610 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:02.960466 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:02.960474 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:02.960478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:02.960499 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:02.960536 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:02.960553 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:02.960678 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:02.960791 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 03:06:02.960844 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 03:06:02.961768 5366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:03.014555 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:03.014001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38947 (local address 127.0.0.1:39204)
0620 03:06:03.014248 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:03.014355 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 03:06:03.015661 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:03.015257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38947 (local address 127.0.0.1:39216)
0620 03:06:03.015447 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:03.015528 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 03:06:03.048496 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 5244
I20260620 03:06:03.066180 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 5374
I20260620 03:06:03.069188 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:03.066948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41946 (local address 127.0.97.3:44267)
0620 03:06:03.067122 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:03.067130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:03.067199 (+ 69us) server_negotiation.cc:407] Connection header received
0620 03:06:03.067458 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:03.067462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:03.067544 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:03.067654 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:03.068555 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:03.068560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:03.068565 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:03.068585 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:03.068620 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:03.068632 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:03.068765 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:03.068851 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 03:06:03.068912 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 03:06:03.069815 5496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:03.076112 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:03.075688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38947 (local address 127.0.0.1:39224)
0620 03:06:03.075862 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:03.075969 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 03:06:03.077062 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:03.076719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38947 (local address 127.0.0.1:39230)
0620 03:06:03.076863 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:03.076950 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 03:06:03.157634 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:03.157092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38947 (local address 127.0.0.1:39234)
0620 03:06:03.157290 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:03.157418 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 03:06:03.158245 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 5374
I20260620 03:06:03.159047 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:03.158478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38947 (local address 127.0.0.1:39240)
0620 03:06:03.158754 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:03.158879 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260620 03:06:03.174708 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 5023
I20260620 03:06:03.176033 5084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:03.282009 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 5023
2026-06-20T03:06:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2542 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 03:06:03.319337 388 mini_oidc.cc:152] Starting JWKS server
I20260620 03:06:03.321918 388 webserver.cc:460] Webserver started at https://127.0.0.1:35011/ using document root <none> and password file <none>
I20260620 03:06:03.322252 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 03:06:03.322930 388 webserver.cc:460] Webserver started at http://127.0.0.1:43901/ using document root <none> and password file <none>
2026-06-20T03:06:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:06:03Z Disabled control of system clock
I20260620 03:06:03.349633 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39795
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39795
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35011/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/testchainca.cert with env {}
W20260620 03:06:03.672101 5556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:03.672472 5556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:03.672637 5556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:03.679785 5556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:06:03.679934 5556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:03.680023 5556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:03.680127 5556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:06:03.680222 5556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 03:06:03.680310 5556 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 03:06:03.680392 5556 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35011/jwks.json
I20260620 03:06:03.690862 5556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39795
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35011/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:03.692296 5556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:03.693616 5556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:03.705595 5561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:03.705482 5562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:03.705482 5564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:03.706396 5556 server_base.cc:1034] running on GCE node
I20260620 03:06:03.707175 5556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:03.708572 5556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:03.709822 5556 hybrid_clock.cc:648] HybridClock initialized: now 1781924763709785 us; error 47 us; skew 500 ppm
I20260620 03:06:03.710198 5556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:06:03.712553 5556 webserver.cc:460] Webserver started at http://127.0.97.62:33281/ using document root <none> and password file <none>
I20260620 03:06:03.713239 5556 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:03.713375 5556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:03.713685 5556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:03.716576 5556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "dec6021910a64f7ca7ca115d60702b0c"
format_stamp: "Formatted at 2026-06-20 03:06:03 on dist-test-slave-xv68"
server_key: "70326b92ecf35303a2ec4d8c50221b11"
server_key_iv: "a7de379b0beaaa75717e660609d53b87"
server_key_version: "encryptionkey@0"
I20260620 03:06:03.717370 5556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "dec6021910a64f7ca7ca115d60702b0c"
format_stamp: "Formatted at 2026-06-20 03:06:03 on dist-test-slave-xv68"
server_key: "70326b92ecf35303a2ec4d8c50221b11"
server_key_iv: "a7de379b0beaaa75717e660609d53b87"
server_key_version: "encryptionkey@0"
I20260620 03:06:03.722396 5556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:06:03.725934 5570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:03.727504 5556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 03:06:03.727698 5556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "dec6021910a64f7ca7ca115d60702b0c"
format_stamp: "Formatted at 2026-06-20 03:06:03 on dist-test-slave-xv68"
server_key: "70326b92ecf35303a2ec4d8c50221b11"
server_key_iv: "a7de379b0beaaa75717e660609d53b87"
server_key_version: "encryptionkey@0"
I20260620 03:06:03.727913 5556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:03.753149 5556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:03.754166 5556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:03.754506 5556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:03.773257 5556 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39795
I20260620 03:06:03.773277 5621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39795 every 8 connection(s)
I20260620 03:06:03.774600 5556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:06:03.776858 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 5556
I20260620 03:06:03.777082 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:06:03.777864 388 external_mini_cluster.cc:1428] Setting key 5a1841b8c6d9792988c667a67a08313b
I20260620 03:06:03.779018 5622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:03.790881 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:03.779867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46094 (local address 127.0.97.62:39795)
0620 03:06:03.780824 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:03.780835 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:03.780873 (+ 38us) server_negotiation.cc:407] Connection header received
0620 03:06:03.782361 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:03.782383 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:03.782845 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:03.783250 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:03.783833 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:03.785127 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:03.786411 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:03.787039 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:03.787228 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:03.788496 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:03.788507 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:03.788519 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:03.789094 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:03.789162 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:03.789184 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:03.789308 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:03.789546 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 03:06:03.789823 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":319,"threads_started":1}
I20260620 03:06:03.791003 5622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c: Bootstrap starting.
I20260620 03:06:03.794507 5622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:03.794764 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:06:03.795636 5622 log.cc:824] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:03.798422 5622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c: No bootstrap required, opened a new log
I20260620 03:06:03.805634 5622 raft_consensus.cc:348] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec6021910a64f7ca7ca115d60702b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39795 } }
I20260620 03:06:03.806030 5622 raft_consensus.cc:374] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:03.806157 5622 raft_consensus.cc:729] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec6021910a64f7ca7ca115d60702b0c, State: Initialized, Role: FOLLOWER
I20260620 03:06:03.806785 5622 consensus_queue.cc:260] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec6021910a64f7ca7ca115d60702b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39795 } }
I20260620 03:06:03.807040 5622 raft_consensus.cc:388] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:06:03.807181 5622 raft_consensus.cc:482] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:06:03.807401 5622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:03.809505 5622 raft_consensus.cc:504] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec6021910a64f7ca7ca115d60702b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39795 } }
I20260620 03:06:03.810001 5622 leader_election.cc:302] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dec6021910a64f7ca7ca115d60702b0c; no voters:
I20260620 03:06:03.810657 5622 leader_election.cc:288] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:06:03.810851 5627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:03.811582 5627 raft_consensus.cc:686] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c [term 1 LEADER]: Becoming Leader. State: Replica: dec6021910a64f7ca7ca115d60702b0c, State: Running, Role: LEADER
I20260620 03:06:03.812273 5627 consensus_queue.cc:237] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec6021910a64f7ca7ca115d60702b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39795 } }
I20260620 03:06:03.813565 5622 sys_catalog.cc:564] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:06:03.815013 5629 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dec6021910a64f7ca7ca115d60702b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec6021910a64f7ca7ca115d60702b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39795 } } }
I20260620 03:06:03.814989 5628 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dec6021910a64f7ca7ca115d60702b0c. Latest consensus state: current_term: 1 leader_uuid: "dec6021910a64f7ca7ca115d60702b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec6021910a64f7ca7ca115d60702b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39795 } } }
I20260620 03:06:03.815439 5629 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:03.815439 5628 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:03.818574 5632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:06:03.823750 5632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:06:03.832882 5632 catalog_manager.cc:1347] Generated new cluster ID: 088fd111465a4f2b85cdb2d9ff533bc2
I20260620 03:06:03.833006 5632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:06:03.843103 5632 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:06:03.844360 5632 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:06:03.850296 5632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dec6021910a64f7ca7ca115d60702b0c: Generated new TSK 0
I20260620 03:06:03.851114 5632 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 03:06:03.885888 5652 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 03:06:03.886075 5652 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 03:06:03.887202 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:03.865452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.0.97.62:39795)
0620 03:06:03.865642 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:03.865651 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:03.866067 (+ 416us) server_negotiation.cc:407] Connection header received
0620 03:06:03.866404 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:03.866409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:03.866477 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:03.866621 (+ 144us) server_negotiation.cc:226] Negotiated authn=JWT
0620 03:06:03.867248 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:03.868456 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:03.870562 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:03.871011 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:03.871145 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:03.871793 (+ 648us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 03:06:03.886481 (+ 14688us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35011/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 03:06:03.886858 (+ 377us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46106: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35011/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 03:06:03.887356 5625 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46106: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35011/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 03:06:03.888733 5651 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:03.865340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39795 (local address 127.0.0.1:46106)
0620 03:06:03.865966 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:03.865982 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:03.866168 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:03.866647 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:03.866652 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:03.866665 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 03:06:03.867044 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:03.867060 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:03.868621 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:03.868627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:03.870415 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:03.870435 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:03.871145 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:03.871220 (+ 75us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:03.871453 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:03.871509 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 03:06:03.886778 (+ 15269us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35011/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 03:06:03.887111 (+ 333us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39795: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35011/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":342,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:39795: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35011/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 03:06:03.890470 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 5556
I20260620 03:06:03.891604 5617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:03.981529 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 5556
2026-06-20T03:06:04Z chronyd exiting
I20260620 03:06:04.031378 388 test_util.cc:175] -----------------------------------------------
I20260620 03:06:04.031546 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924720365844-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (716 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 03:06:04.035157 388 mini_oidc.cc:152] Starting JWKS server
I20260620 03:06:04.036979 388 webserver.cc:460] Webserver started at https://127.0.0.1:42103/ using document root <none> and password file <none>
I20260620 03:06:04.037220 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 03:06:04.037854 388 webserver.cc:460] Webserver started at http://127.0.0.1:40105/ using document root <none> and password file <none>
2026-06-20T03:06:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:06:04Z Disabled control of system clock
I20260620 03:06:04.063997 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41777
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41777
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42103/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/testchainca.cert with env {}
W20260620 03:06:04.381870 5659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:04.382180 5659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:04.382267 5659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:04.389550 5659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:06:04.389664 5659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:04.389703 5659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:04.389752 5659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:06:04.389797 5659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 03:06:04.389863 5659 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 03:06:04.389904 5659 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42103/jwks.json
I20260620 03:06:04.399952 5659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41777
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42103/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:04.401192 5659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:04.402431 5659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:04.413635 5664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:04.413758 5665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:04.413900 5667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:04.415033 5659 server_base.cc:1034] running on GCE node
I20260620 03:06:04.415812 5659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:04.417025 5659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:04.418215 5659 hybrid_clock.cc:648] HybridClock initialized: now 1781924764418184 us; error 30 us; skew 500 ppm
I20260620 03:06:04.418519 5659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:06:04.420684 5659 webserver.cc:460] Webserver started at http://127.0.97.62:44491/ using document root <none> and password file <none>
I20260620 03:06:04.421295 5659 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:04.421380 5659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:04.421625 5659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:04.424321 5659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "dd2f568f2f2d4445a8098a3a970cf38e"
format_stamp: "Formatted at 2026-06-20 03:06:04 on dist-test-slave-xv68"
server_key: "eb2682e49d356905ad4bc34f01c1d168"
server_key_iv: "f06718a34deaf1466377696450a83c3a"
server_key_version: "encryptionkey@0"
I20260620 03:06:04.425051 5659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "dd2f568f2f2d4445a8098a3a970cf38e"
format_stamp: "Formatted at 2026-06-20 03:06:04 on dist-test-slave-xv68"
server_key: "eb2682e49d356905ad4bc34f01c1d168"
server_key_iv: "f06718a34deaf1466377696450a83c3a"
server_key_version: "encryptionkey@0"
I20260620 03:06:04.429968 5659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:06:04.433472 5673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:04.434844 5659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:06:04.435030 5659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "dd2f568f2f2d4445a8098a3a970cf38e"
format_stamp: "Formatted at 2026-06-20 03:06:04 on dist-test-slave-xv68"
server_key: "eb2682e49d356905ad4bc34f01c1d168"
server_key_iv: "f06718a34deaf1466377696450a83c3a"
server_key_version: "encryptionkey@0"
I20260620 03:06:04.435279 5659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:04.454741 5659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:04.455590 5659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:04.455866 5659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:04.474120 5659 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41777
I20260620 03:06:04.474143 5724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41777 every 8 connection(s)
I20260620 03:06:04.475447 5659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:06:04.479117 5725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:04.481380 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 5659
I20260620 03:06:04.481567 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:06:04.482347 388 external_mini_cluster.cc:1428] Setting key c10ca8ceb71f432f8761e9652bebfb42
I20260620 03:06:04.491438 5725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e: Bootstrap starting.
I20260620 03:06:04.494987 5725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:04.495919 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:04.484221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48138 (local address 127.0.97.62:41777)
0620 03:06:04.485240 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:04.485251 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:04.485290 (+ 39us) server_negotiation.cc:407] Connection header received
0620 03:06:04.486893 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:04.486915 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:04.487406 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:04.487720 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:04.488262 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:04.489544 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:04.491267 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:04.492110 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:04.492324 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:04.493519 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:04.493527 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:04.493540 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:04.494079 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:04.494167 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:04.494188 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:04.494321 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:04.494520 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 03:06:04.494857 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":360,"threads_started":1}
I20260620 03:06:04.496102 5725 log.cc:824] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:04.499342 5725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e: No bootstrap required, opened a new log
I20260620 03:06:04.500160 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:06:04.507684 5725 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2f568f2f2d4445a8098a3a970cf38e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41777 } }
I20260620 03:06:04.508061 5725 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:04.508201 5725 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd2f568f2f2d4445a8098a3a970cf38e, State: Initialized, Role: FOLLOWER
I20260620 03:06:04.508740 5725 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2f568f2f2d4445a8098a3a970cf38e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41777 } }
I20260620 03:06:04.508939 5725 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:06:04.509030 5725 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:06:04.509145 5725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:04.511149 5725 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2f568f2f2d4445a8098a3a970cf38e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41777 } }
I20260620 03:06:04.511670 5725 leader_election.cc:302] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd2f568f2f2d4445a8098a3a970cf38e; no voters:
I20260620 03:06:04.512369 5725 leader_election.cc:288] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:06:04.512494 5730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:04.513232 5730 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e [term 1 LEADER]: Becoming Leader. State: Replica: dd2f568f2f2d4445a8098a3a970cf38e, State: Running, Role: LEADER
I20260620 03:06:04.513707 5730 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2f568f2f2d4445a8098a3a970cf38e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41777 } }
I20260620 03:06:04.514315 5725 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:06:04.516731 5732 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd2f568f2f2d4445a8098a3a970cf38e. Latest consensus state: current_term: 1 leader_uuid: "dd2f568f2f2d4445a8098a3a970cf38e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2f568f2f2d4445a8098a3a970cf38e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41777 } } }
I20260620 03:06:04.516700 5731 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd2f568f2f2d4445a8098a3a970cf38e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2f568f2f2d4445a8098a3a970cf38e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41777 } } }
I20260620 03:06:04.517935 5731 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:04.518362 5732 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:04.520282 5738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:06:04.525313 5738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:06:04.533552 5738 catalog_manager.cc:1347] Generated new cluster ID: 876c746991434d82a8adfcc13a9b9707
I20260620 03:06:04.533663 5738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:06:04.545441 5738 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:06:04.546583 5738 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:06:04.552407 5738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd2f568f2f2d4445a8098a3a970cf38e: Generated new TSK 0
I20260620 03:06:04.553263 5738 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 03:06:04.592880 5755 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 03:06:04.593003 5755 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 03:06:04.594167 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:04.575264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48154 (local address 127.0.97.62:41777)
0620 03:06:04.575464 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:04.575473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:04.575938 (+ 465us) server_negotiation.cc:407] Connection header received
0620 03:06:04.576175 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:04.576179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:04.576244 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:04.576385 (+ 141us) server_negotiation.cc:226] Negotiated authn=JWT
0620 03:06:04.576857 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:04.577578 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:04.578927 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:04.579425 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:04.579552 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:04.580016 (+ 464us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 03:06:04.593489 (+ 13473us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 03:06:04.593864 (+ 375us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48154: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 03:06:04.594290 5728 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48154: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 03:06:04.594323 5754 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:04.575211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41777 (local address 127.0.0.1:48154)
0620 03:06:04.575829 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:04.575847 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:04.576000 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:04.576400 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:04.576404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:04.576426 (+ 22us) client_negotiation.cc:189] Negotiated authn=JWT
0620 03:06:04.576697 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:04.576705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:04.577727 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:04.577733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:04.578777 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:04.578788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:04.579542 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:04.579588 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:04.579738 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:04.579771 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 03:06:04.593708 (+ 13937us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 03:06:04.593941 (+ 233us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:41777: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":346,"threads_started":1}
I20260620 03:06:04.595850 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 5659
I20260620 03:06:04.596946 5720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:04.687711 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 5659
2026-06-20T03:06:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (709 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 03:06:04.744549 388 mini_oidc.cc:152] Starting JWKS server
I20260620 03:06:04.746387 388 webserver.cc:460] Webserver started at https://127.0.0.1:43435/ using document root <none> and password file <none>
I20260620 03:06:04.746644 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 03:06:04.747284 388 webserver.cc:460] Webserver started at http://127.0.0.1:41619/ using document root <none> and password file <none>
2026-06-20T03:06:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:06:04Z Disabled control of system clock
I20260620 03:06:04.773290 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36653
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36653
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43435/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 03:06:05.111339 5762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:05.111639 5762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:05.111724 5762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:05.118778 5762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:06:05.118885 5762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:05.118923 5762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:05.118973 5762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:06:05.119017 5762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 03:06:05.119071 5762 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 03:06:05.119112 5762 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43435/jwks.json
I20260620 03:06:05.128116 5762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36653
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43435/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:05.129351 5762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:05.131098 5762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:05.142993 5767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:05.142962 5768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:05.143051 5770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:05.144181 5762 server_base.cc:1034] running on GCE node
I20260620 03:06:05.144882 5762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:05.146183 5762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:05.147399 5762 hybrid_clock.cc:648] HybridClock initialized: now 1781924765147363 us; error 25 us; skew 500 ppm
I20260620 03:06:05.147756 5762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:06:05.150082 5762 webserver.cc:460] Webserver started at http://127.0.97.62:44361/ using document root <none> and password file <none>
I20260620 03:06:05.150804 5762 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:05.150916 5762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:05.151252 5762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:05.154101 5762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "c51c732b52044025926d6265cf502462"
format_stamp: "Formatted at 2026-06-20 03:06:05 on dist-test-slave-xv68"
server_key: "c8643587a37669cca46f7ca2c083c73d"
server_key_iv: "ce7804f99f2c26ef7551c4b7dfb46d10"
server_key_version: "encryptionkey@0"
I20260620 03:06:05.154879 5762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "c51c732b52044025926d6265cf502462"
format_stamp: "Formatted at 2026-06-20 03:06:05 on dist-test-slave-xv68"
server_key: "c8643587a37669cca46f7ca2c083c73d"
server_key_iv: "ce7804f99f2c26ef7551c4b7dfb46d10"
server_key_version: "encryptionkey@0"
I20260620 03:06:05.159837 5762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 03:06:05.163210 5776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:05.164597 5762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:06:05.164785 5762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "c51c732b52044025926d6265cf502462"
format_stamp: "Formatted at 2026-06-20 03:06:05 on dist-test-slave-xv68"
server_key: "c8643587a37669cca46f7ca2c083c73d"
server_key_iv: "ce7804f99f2c26ef7551c4b7dfb46d10"
server_key_version: "encryptionkey@0"
I20260620 03:06:05.165004 5762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:05.176272 5762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:05.177054 5762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:05.177346 5762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:05.196496 5762 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36653
I20260620 03:06:05.196519 5827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36653 every 8 connection(s)
I20260620 03:06:05.197842 5762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:06:05.201545 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 5762
I20260620 03:06:05.201778 5828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:05.201788 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:06:05.202425 388 external_mini_cluster.cc:1428] Setting key e24e1fad895c43e68e455688eaa9ed17
I20260620 03:06:05.216035 5828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462: Bootstrap starting.
I20260620 03:06:05.216403 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:05.204499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46972 (local address 127.0.97.62:36653)
0620 03:06:05.205556 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:05.205567 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:05.205606 (+ 39us) server_negotiation.cc:407] Connection header received
0620 03:06:05.207026 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:05.207048 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:05.207783 (+ 735us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:05.208128 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:05.208731 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:05.210073 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:05.211834 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:05.212526 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:05.212685 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:05.213935 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:05.213943 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:05.213960 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:05.214513 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:05.214587 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:05.214607 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:05.214696 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:05.214933 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 03:06:05.215282 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":403,"threads_started":1}
I20260620 03:06:05.219604 5828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:05.220664 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:06:05.220885 5828 log.cc:824] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:05.224105 5828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462: No bootstrap required, opened a new log
I20260620 03:06:05.231592 5828 raft_consensus.cc:348] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51c732b52044025926d6265cf502462" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36653 } }
I20260620 03:06:05.231951 5828 raft_consensus.cc:374] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:05.232064 5828 raft_consensus.cc:729] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c51c732b52044025926d6265cf502462, State: Initialized, Role: FOLLOWER
I20260620 03:06:05.232620 5828 consensus_queue.cc:260] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51c732b52044025926d6265cf502462" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36653 } }
I20260620 03:06:05.232843 5828 raft_consensus.cc:388] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:06:05.232998 5828 raft_consensus.cc:482] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:06:05.233160 5828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:05.235595 5828 raft_consensus.cc:504] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51c732b52044025926d6265cf502462" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36653 } }
I20260620 03:06:05.236093 5828 leader_election.cc:302] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c51c732b52044025926d6265cf502462; no voters:
I20260620 03:06:05.236704 5828 leader_election.cc:288] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:06:05.236876 5833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:05.237632 5833 raft_consensus.cc:686] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462 [term 1 LEADER]: Becoming Leader. State: Replica: c51c732b52044025926d6265cf502462, State: Running, Role: LEADER
I20260620 03:06:05.238198 5833 consensus_queue.cc:237] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51c732b52044025926d6265cf502462" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36653 } }
I20260620 03:06:05.238792 5828 sys_catalog.cc:564] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:06:05.240937 5834 sys_catalog.cc:455] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c51c732b52044025926d6265cf502462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51c732b52044025926d6265cf502462" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36653 } } }
I20260620 03:06:05.241232 5834 sys_catalog.cc:458] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:05.241052 5835 sys_catalog.cc:455] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c51c732b52044025926d6265cf502462. Latest consensus state: current_term: 1 leader_uuid: "c51c732b52044025926d6265cf502462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51c732b52044025926d6265cf502462" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36653 } } }
I20260620 03:06:05.241525 5835 sys_catalog.cc:458] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:05.242280 5839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:06:05.250145 5839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:06:05.261433 5839 catalog_manager.cc:1347] Generated new cluster ID: bda955a188994fa7876312d14eb71d7c
I20260620 03:06:05.261575 5839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:06:05.273954 5839 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:06:05.275259 5839 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:06:05.282012 5839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c51c732b52044025926d6265cf502462: Generated new TSK 0
I20260620 03:06:05.282995 5839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:06:05.306143 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:05.297611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.0.97.62:36653)
0620 03:06:05.297811 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:05.297817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:05.298223 (+ 406us) server_negotiation.cc:407] Connection header received
0620 03:06:05.298419 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:05.298424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:05.298517 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:05.298691 (+ 174us) server_negotiation.cc:226] Negotiated authn=JWT
0620 03:06:05.299112 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:05.300315 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:05.301790 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:05.302485 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:05.302620 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:05.303097 (+ 477us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 03:06:05.305498 (+ 2401us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43435/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 03:06:05.305843 (+ 345us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46976: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43435/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 03:06:05.306272 5831 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46976: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43435/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 03:06:05.307776 5857 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:05.297483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36653 (local address 127.0.0.1:46976)
0620 03:06:05.298113 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:05.298130 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:05.298248 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:05.298702 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:05.298705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:05.298714 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 03:06:05.298974 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:05.298981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:05.300507 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:05.300516 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:05.301644 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:05.301656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:05.302613 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:05.302674 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:05.302794 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:05.302826 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 03:06:05.305703 (+ 2877us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43435/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 03:06:05.307357 (+ 1654us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:36653: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43435/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":1317,"thread_start_us":333,"threads_started":1}
I20260620 03:06:05.308801 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 5762
I20260620 03:06:05.309859 5823 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 03:06:05.317369 5858 webserver.cc:533] Webserver: sslize failed:
E20260620 03:06:05.317487 5858 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 03:06:05.401901 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 5762
2026-06-20T03:06:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (712 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T03:06:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:06:05Z Disabled control of system clock
I20260620 03:06:05.483078 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43251
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43251
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/testchainca.cert with env {}
W20260620 03:06:05.802820 5863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:05.803082 5863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:05.803256 5863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:05.810325 5863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:06:05.810423 5863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:05.810505 5863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:05.810592 5863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:06:05.810647 5863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 03:06:05.810748 5863 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 03:06:05.810808 5863 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 03:06:05.820362 5863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:05.821599 5863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:05.822853 5863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:05.834371 5868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:05.834393 5869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:05.834901 5871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:05.835227 5863 server_base.cc:1034] running on GCE node
I20260620 03:06:05.835990 5863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:05.837390 5863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:05.838608 5863 hybrid_clock.cc:648] HybridClock initialized: now 1781924765838552 us; error 55 us; skew 500 ppm
I20260620 03:06:05.838945 5863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:06:05.841230 5863 webserver.cc:460] Webserver started at http://127.0.97.62:34949/ using document root <none> and password file <none>
I20260620 03:06:05.841893 5863 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:05.842022 5863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:05.842338 5863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:05.845150 5863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "d543697462fe481294ac9512bf872f99"
format_stamp: "Formatted at 2026-06-20 03:06:05 on dist-test-slave-xv68"
server_key: "7939459ea24562671bf369f478407dd9"
server_key_iv: "01040597ea8506d617dc9ffbe32dec89"
server_key_version: "encryptionkey@0"
I20260620 03:06:05.845930 5863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "d543697462fe481294ac9512bf872f99"
format_stamp: "Formatted at 2026-06-20 03:06:05 on dist-test-slave-xv68"
server_key: "7939459ea24562671bf369f478407dd9"
server_key_iv: "01040597ea8506d617dc9ffbe32dec89"
server_key_version: "encryptionkey@0"
I20260620 03:06:05.850917 5863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:06:05.854517 5877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:05.855950 5863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:06:05.856132 5863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "d543697462fe481294ac9512bf872f99"
format_stamp: "Formatted at 2026-06-20 03:06:05 on dist-test-slave-xv68"
server_key: "7939459ea24562671bf369f478407dd9"
server_key_iv: "01040597ea8506d617dc9ffbe32dec89"
server_key_version: "encryptionkey@0"
I20260620 03:06:05.856359 5863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:05.877537 5863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:05.878371 5863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:05.878662 5863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:05.896759 5863 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43251
I20260620 03:06:05.896793 5928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43251 every 8 connection(s)
I20260620 03:06:05.898082 5863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:06:05.901737 5929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:05.902385 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 5863
I20260620 03:06:05.902616 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:06:05.903271 388 external_mini_cluster.cc:1428] Setting key 53136fb4886f484d31d943de526a57f3
I20260620 03:06:05.914048 5929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99: Bootstrap starting.
I20260620 03:06:05.916458 5932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:05.905118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57518 (local address 127.0.97.62:43251)
0620 03:06:05.906229 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:05.906244 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:05.906303 (+ 59us) server_negotiation.cc:407] Connection header received
0620 03:06:05.908137 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:05.908158 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:05.908610 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:05.909008 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:05.909553 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:05.910781 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:05.912135 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:05.912795 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:05.913002 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:05.914209 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:05.914222 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:05.914236 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:05.914757 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:05.914850 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:05.914870 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:05.915001 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:05.915237 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 03:06:05.915560 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":352,"threads_started":1}
I20260620 03:06:05.917201 5929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:05.918218 5929 log.cc:824] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:05.920336 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:06:05.921023 5929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99: No bootstrap required, opened a new log
I20260620 03:06:05.928654 5929 raft_consensus.cc:348] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d543697462fe481294ac9512bf872f99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43251 } }
I20260620 03:06:05.929025 5929 raft_consensus.cc:374] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:05.929133 5929 raft_consensus.cc:729] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d543697462fe481294ac9512bf872f99, State: Initialized, Role: FOLLOWER
I20260620 03:06:05.929666 5929 consensus_queue.cc:260] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d543697462fe481294ac9512bf872f99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43251 } }
I20260620 03:06:05.929896 5929 raft_consensus.cc:388] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:06:05.930042 5929 raft_consensus.cc:482] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:06:05.930200 5929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:05.932456 5929 raft_consensus.cc:504] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d543697462fe481294ac9512bf872f99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43251 } }
I20260620 03:06:05.932948 5929 leader_election.cc:302] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d543697462fe481294ac9512bf872f99; no voters:
I20260620 03:06:05.933559 5929 leader_election.cc:288] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:06:05.933751 5934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:05.934866 5934 raft_consensus.cc:686] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99 [term 1 LEADER]: Becoming Leader. State: Replica: d543697462fe481294ac9512bf872f99, State: Running, Role: LEADER
I20260620 03:06:05.935498 5934 consensus_queue.cc:237] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d543697462fe481294ac9512bf872f99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43251 } }
I20260620 03:06:05.935886 5929 sys_catalog.cc:564] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:06:05.938364 5936 sys_catalog.cc:455] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d543697462fe481294ac9512bf872f99. Latest consensus state: current_term: 1 leader_uuid: "d543697462fe481294ac9512bf872f99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d543697462fe481294ac9512bf872f99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43251 } } }
I20260620 03:06:05.938814 5936 sys_catalog.cc:458] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:05.939393 5935 sys_catalog.cc:455] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d543697462fe481294ac9512bf872f99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d543697462fe481294ac9512bf872f99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43251 } } }
I20260620 03:06:05.939834 5935 sys_catalog.cc:458] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:05.940619 5943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:06:05.946398 5943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:06:05.955507 5943 catalog_manager.cc:1347] Generated new cluster ID: c567cd1debb347e18388499f3081dd03
I20260620 03:06:05.955624 5943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:06:05.987672 5943 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:06:05.988984 5943 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:06:05.995821 5943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d543697462fe481294ac9512bf872f99: Generated new TSK 0
I20260620 03:06:05.996923 5943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:06:06.494530 5932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:06.019784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57526 (local address 127.0.97.62:43251)
0620 03:06:06.020006 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:06.020012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:06.020395 (+ 383us) server_negotiation.cc:407] Connection header received
0620 03:06:06.020606 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:06.020612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:06.020697 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:06.020874 (+ 177us) server_negotiation.cc:226] Negotiated authn=JWT
0620 03:06:06.021266 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:06.022445 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:06.023873 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:06.024553 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:06.024671 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:06.025117 (+ 446us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 03:06:06.493938 (+468821us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 03:06:06.494260 (+ 322us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57526: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 03:06:06.494732 5932 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57526: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 03:06:06.494959 5958 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:06.019662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43251 (local address 127.0.0.1:57526)
0620 03:06:06.020297 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:06.020314 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:06.020428 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:06.020886 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:06.020889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:06.020897 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 03:06:06.021132 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:06.021139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:06.022602 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:06.022606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:06.023708 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:06.023719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:06.024673 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:06.024715 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:06.024850 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:06.024879 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 03:06:06.494234 (+469355us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 03:06:06.494478 (+ 244us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:43251: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":349,"threads_started":1}
I20260620 03:06:06.496546 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 5863
I20260620 03:06:06.497501 5924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:06.596961 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 5863
2026-06-20T03:06:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1176 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (402 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (406 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:06:07 dist-test-slave-xv68 krb5kdc[5973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:06:07 dist-test-slave-xv68 krb5kdc[5973](info): set up 2 sockets
Jun 20 03:06:07 dist-test-slave-xv68 krb5kdc[5973](info): commencing operation
krb5kdc: starting...
W20260620 03:06:08.499961 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.008s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:06:08 dist-test-slave-xv68 krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:06:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:06:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:06:08.703866 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34615
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:09.025166 5986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:09.025472 5986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:09.025655 5986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:09.032907 5986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:06:09.033020 5986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:09.033107 5986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:09.033196 5986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:06:09.033279 5986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:06:09.043077 5986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:09.044430 5986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:09.045718 5986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:09.057292 5991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:09.057225 5994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:09.057266 5992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:09.058073 5986 server_base.cc:1034] running on GCE node
I20260620 03:06:09.058856 5986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:09.060312 5986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:09.061540 5986 hybrid_clock.cc:648] HybridClock initialized: now 1781924769061488 us; error 51 us; skew 500 ppm
I20260620 03:06:09.061885 5986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:09 dist-test-slave-xv68 krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:09.065121 5986 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:06:09.066555 5986 webserver.cc:460] Webserver started at http://127.0.97.62:39701/ using document root <none> and password file <none>
I20260620 03:06:09.067233 5986 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:09.067428 5986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:09.067741 5986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:09.070665 5986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "69b4b56c433b4e5f909312caa3e0d672"
format_stamp: "Formatted at 2026-06-20 03:06:09 on dist-test-slave-xv68"
server_key: "9c18255b962d5d48224d928a933e5ca6"
server_key_iv: "c5a829337683c34e3258b0d650ea6085"
server_key_version: "encryptionkey@0"
I20260620 03:06:09.071516 5986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "69b4b56c433b4e5f909312caa3e0d672"
format_stamp: "Formatted at 2026-06-20 03:06:09 on dist-test-slave-xv68"
server_key: "9c18255b962d5d48224d928a933e5ca6"
server_key_iv: "c5a829337683c34e3258b0d650ea6085"
server_key_version: "encryptionkey@0"
I20260620 03:06:09.076535 5986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:06:09.080020 6001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:09.081293 5986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:06:09.081470 5986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "69b4b56c433b4e5f909312caa3e0d672"
format_stamp: "Formatted at 2026-06-20 03:06:09 on dist-test-slave-xv68"
server_key: "9c18255b962d5d48224d928a933e5ca6"
server_key_iv: "c5a829337683c34e3258b0d650ea6085"
server_key_version: "encryptionkey@0"
I20260620 03:06:09.081666 5986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:09.101797 5986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:09.106406 5986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:09.106696 5986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:09.125399 5986 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34615
I20260620 03:06:09.125428 6052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34615 every 8 connection(s)
I20260620 03:06:09.126760 5986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:06:09.130486 6053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:09.133966 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 5986
I20260620 03:06:09.134177 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:06:09.134800 388 external_mini_cluster.cc:1428] Setting key b6320f71bc0777620867b8a0b914768c
I20260620 03:06:09.143146 6053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672: Bootstrap starting.
I20260620 03:06:09.146821 6053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:09.148164 6053 log.cc:824] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672: Log is configured to *not* fsync() on all Append() calls
Jun 20 03:06:09 dist-test-slave-xv68 krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:06:09.151360 6053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672: No bootstrap required, opened a new log
I20260620 03:06:09.158481 6056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:09.136729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52580 (local address 127.0.97.62:34615)
0620 03:06:09.137493 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:09.137504 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:09.137545 (+ 41us) server_negotiation.cc:407] Connection header received
0620 03:06:09.139038 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:09.139059 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:09.139515 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:09.139838 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:09.141637 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:09.142971 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:09.144662 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:09.145325 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:09.145461 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:09.149983 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:09.149994 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:09.150012 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:09.150056 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:09.153947 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:09.154570 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:09.154579 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:09.154587 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:09.154792 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:09.155117 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:09.155122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:09.155125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:09.155832 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:09.156134 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:09.156423 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 03:06:09.156896 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":373,"threads_started":1}
I20260620 03:06:09.162087 6053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b4b56c433b4e5f909312caa3e0d672" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34615 } }
I20260620 03:06:09.162433 6053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:09.162498 6053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69b4b56c433b4e5f909312caa3e0d672, State: Initialized, Role: FOLLOWER
I20260620 03:06:09.162998 6053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b4b56c433b4e5f909312caa3e0d672" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34615 } }
I20260620 03:06:09.163184 6053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:06:09.163275 6053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:06:09.163477 6053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:09.165555 6053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b4b56c433b4e5f909312caa3e0d672" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34615 } }
I20260620 03:06:09.165990 6053 leader_election.cc:302] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69b4b56c433b4e5f909312caa3e0d672; no voters:
I20260620 03:06:09.166570 6053 leader_election.cc:288] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:06:09.166754 6058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:09.167471 6058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672 [term 1 LEADER]: Becoming Leader. State: Replica: 69b4b56c433b4e5f909312caa3e0d672, State: Running, Role: LEADER
I20260620 03:06:09.168105 6058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b4b56c433b4e5f909312caa3e0d672" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34615 } }
I20260620 03:06:09.168889 6053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:06:09.170917 6059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69b4b56c433b4e5f909312caa3e0d672" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b4b56c433b4e5f909312caa3e0d672" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34615 } } }
I20260620 03:06:09.171367 6059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:09.170955 6060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69b4b56c433b4e5f909312caa3e0d672. Latest consensus state: current_term: 1 leader_uuid: "69b4b56c433b4e5f909312caa3e0d672" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b4b56c433b4e5f909312caa3e0d672" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34615 } } }
I20260620 03:06:09.171536 6060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:09.172525 6064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:06:09.182678 6064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:06:09.191686 6064 catalog_manager.cc:1347] Generated new cluster ID: eeba7464ab0f4d0f99cb89ffb42b9ee7
I20260620 03:06:09.191813 6064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:06:09.199095 6064 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:06:09.201113 6064 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:06:09.220264 6064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69b4b56c433b4e5f909312caa3e0d672: Generated new TSK 0
I20260620 03:06:09.221345 6064 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:06:09.271256 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34615
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:09.597625 6079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:09.597934 6079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:09.598109 6079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:09.605468 6079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:09.605604 6079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:09.605782 6079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:06:09.615319 6079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:09.616552 6079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:09.617816 6079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:09.630748 6085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:09.630846 6084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:09.630846 6087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:09.631974 6079 server_base.cc:1034] running on GCE node
I20260620 03:06:09.632624 6079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:09.633632 6079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:09.634840 6079 hybrid_clock.cc:648] HybridClock initialized: now 1781924769634778 us; error 45 us; skew 500 ppm
I20260620 03:06:09.635170 6079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:09 dist-test-slave-xv68 krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:09.638350 6079 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:06:09.639765 6079 webserver.cc:460] Webserver started at http://127.0.97.1:36477/ using document root <none> and password file <none>
I20260620 03:06:09.640380 6079 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:09.640479 6079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:09.640730 6079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:09.643633 6079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "3dc7b0f64c6d430898573e2d5d197f3b"
format_stamp: "Formatted at 2026-06-20 03:06:09 on dist-test-slave-xv68"
server_key: "6c77f3fa3349bb7ce391c2b9d360c5f4"
server_key_iv: "99c83dee72bb476fe7d9094b4f33ed92"
server_key_version: "encryptionkey@0"
I20260620 03:06:09.644335 6079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3dc7b0f64c6d430898573e2d5d197f3b"
format_stamp: "Formatted at 2026-06-20 03:06:09 on dist-test-slave-xv68"
server_key: "6c77f3fa3349bb7ce391c2b9d360c5f4"
server_key_iv: "99c83dee72bb476fe7d9094b4f33ed92"
server_key_version: "encryptionkey@0"
I20260620 03:06:09.649112 6079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 03:06:09.652386 6094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:09.653745 6079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:06:09.653941 6079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "3dc7b0f64c6d430898573e2d5d197f3b"
format_stamp: "Formatted at 2026-06-20 03:06:09 on dist-test-slave-xv68"
server_key: "6c77f3fa3349bb7ce391c2b9d360c5f4"
server_key_iv: "99c83dee72bb476fe7d9094b4f33ed92"
server_key_version: "encryptionkey@0"
I20260620 03:06:09.654129 6079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:09.678638 6079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:09.683089 6079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:09.683454 6079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:09.684396 6079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:09.685834 6079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:09.685940 6079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:09.686048 6079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:09.686097 6079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:09.718936 6079 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41933
I20260620 03:06:09.719033 6206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41933 every 8 connection(s)
I20260620 03:06:09.720249 6079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:06:09.723001 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 6079
I20260620 03:06:09.723176 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:06:09.724006 388 external_mini_cluster.cc:1428] Setting key 465dd9d019639156c9bbe893f94aefde
Jun 20 03:06:09 dist-test-slave-xv68 krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:06:09.740039 6209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:09.722808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46159 (local address 127.0.97.62:34615)
0620 03:06:09.723278 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:09.723325 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:09.724522 (+ 1197us) server_negotiation.cc:407] Connection header received
0620 03:06:09.726497 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:09.726503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:09.726582 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:09.726713 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:09.728302 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:09.729040 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:09.730567 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:09.730985 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:09.731124 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:09.734553 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:09.734558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:09.734561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:09.734590 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:09.737062 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:09.737774 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:09.737784 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:09.737786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:09.737893 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:09.738332 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:09.738337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:09.738340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:09.738588 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:09.738742 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:09.739441 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 03:06:09.739662 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":249,"threads_started":1}
I20260620 03:06:09.740986 6210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:09.723529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34615 (local address 127.0.97.1:46159)
0620 03:06:09.724360 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:09.724403 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:09.726230 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:09.726948 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:09.726957 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:09.727554 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:09.728118 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:09.728152 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:09.729201 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:09.729205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:09.730419 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:09.730432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:09.731209 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:09.731219 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:09.731496 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:09.732181 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:09.732203 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:09.734323 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:09.737203 (+ 2880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:09.737212 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:09.737228 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:09.737646 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:09.738018 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:09.738023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:09.738026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:09.738215 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:09.738720 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:09.738728 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:09.739051 (+ 323us) client_negotiation.cc:769] Sending connection context
0620 03:06:09.739305 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 03:06:09.739841 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":342,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 03:06:09.742616 6207 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34615
I20260620 03:06:09.742878 6207 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:09.743659 6207 heartbeater.cc:511] Master 127.0.97.62:34615 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 03:06:09.745769 6018 ts_manager.cc:194] Registered new tserver with Master: 3dc7b0f64c6d430898573e2d5d197f3b (127.0.97.1:41933)
I20260620 03:06:09.747162 6018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:06:09.769977 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34615
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:10.087702 6213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:10.087971 6213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:10.088063 6213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:10.095389 6213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:10.095507 6213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:10.095620 6213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:06:10.105106 6213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:10.106292 6213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:10.107583 6213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:10.120209 6219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:10.120298 6221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:10.120452 6218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:10.121176 6213 server_base.cc:1034] running on GCE node
I20260620 03:06:10.121760 6213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:10.122694 6213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:10.123926 6213 hybrid_clock.cc:648] HybridClock initialized: now 1781924770123884 us; error 75 us; skew 500 ppm
I20260620 03:06:10.124245 6213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:10 dist-test-slave-xv68 krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:10.132340 6213 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:06:10.133970 6213 webserver.cc:460] Webserver started at http://127.0.97.2:32931/ using document root <none> and password file <none>
I20260620 03:06:10.134657 6213 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:10.134805 6213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:10.135106 6213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:10.137974 6213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "38ad128522e54cb39688154533c306b9"
format_stamp: "Formatted at 2026-06-20 03:06:10 on dist-test-slave-xv68"
server_key: "334250d69d558fbd8e7c03daed63fb21"
server_key_iv: "9cbc31dab1ac9f387eb79a7f1522ed5e"
server_key_version: "encryptionkey@0"
I20260620 03:06:10.138747 6213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "38ad128522e54cb39688154533c306b9"
format_stamp: "Formatted at 2026-06-20 03:06:10 on dist-test-slave-xv68"
server_key: "334250d69d558fbd8e7c03daed63fb21"
server_key_iv: "9cbc31dab1ac9f387eb79a7f1522ed5e"
server_key_version: "encryptionkey@0"
I20260620 03:06:10.143743 6213 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 03:06:10.146946 6228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:10.148273 6213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:06:10.148454 6213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "38ad128522e54cb39688154533c306b9"
format_stamp: "Formatted at 2026-06-20 03:06:10 on dist-test-slave-xv68"
server_key: "334250d69d558fbd8e7c03daed63fb21"
server_key_iv: "9cbc31dab1ac9f387eb79a7f1522ed5e"
server_key_version: "encryptionkey@0"
I20260620 03:06:10.148658 6213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:10.163065 6213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:10.167233 6213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:10.167538 6213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:10.168471 6213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:10.169795 6213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:10.169931 6213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:10.170073 6213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:10.170148 6213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:10.203383 6213 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44575
I20260620 03:06:10.203474 6340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44575 every 8 connection(s)
I20260620 03:06:10.204720 6213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:06:10.209201 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 6213
I20260620 03:06:10.209360 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:06:10.209776 388 external_mini_cluster.cc:1428] Setting key 19687afcb77fa597a45629f0c749d10b
Jun 20 03:06:10 dist-test-slave-xv68 krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:06:10.225822 6209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:10.207202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32899 (local address 127.0.97.62:34615)
0620 03:06:10.207422 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:10.207428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:10.208569 (+ 1141us) server_negotiation.cc:407] Connection header received
0620 03:06:10.210466 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:10.210470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:10.210570 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:10.210679 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:10.212537 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:10.213740 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:10.215249 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:10.216008 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:10.216146 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:10.219723 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:10.219728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:10.219732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:10.219760 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:10.222458 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:10.223419 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:10.223423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:10.223425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:10.223548 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:10.224022 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:10.224025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:10.224027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:10.224270 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:10.224376 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:10.225216 (+ 840us) server_negotiation.cc:299] Negotiation successful
0620 03:06:10.225383 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 03:06:10.227073 6343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:10.207720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34615 (local address 127.0.97.2:32899)
0620 03:06:10.208409 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:10.208451 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:10.210162 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:10.210965 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:10.210975 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:10.211735 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:10.212364 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:10.212381 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:10.213886 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:10.213890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:10.215102 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:10.215113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:10.216193 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:10.216197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:10.216384 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:10.217061 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:10.217082 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:10.219468 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:10.222592 (+ 3124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:10.222604 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:10.222622 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:10.223236 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:10.223661 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:10.223669 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:10.223675 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:10.223905 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:10.224399 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:10.224409 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:10.224830 (+ 421us) client_negotiation.cc:769] Sending connection context
0620 03:06:10.225154 (+ 324us) client_negotiation.cc:240] Negotiation successful
0620 03:06:10.225694 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":273,"threads_started":1}
I20260620 03:06:10.228543 6341 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34615
I20260620 03:06:10.228881 6341 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:10.229830 6341 heartbeater.cc:511] Master 127.0.97.62:34615 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 03:06:10.231521 6018 ts_manager.cc:194] Registered new tserver with Master: 38ad128522e54cb39688154533c306b9 (127.0.97.2:44575)
I20260620 03:06:10.232342 6018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:32899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:06:10.256732 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34615
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:10.573308 6346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:10.573627 6346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:10.573796 6346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:10.580729 6346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:10.581070 6346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:10.581292 6346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:06:10.591112 6346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:10.592500 6346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:10.593779 6346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:10.606874 6351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:10.606828 6354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:10.606920 6352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:10.607834 6346 server_base.cc:1034] running on GCE node
I20260620 03:06:10.608424 6346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:10.609324 6346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:10.610551 6346 hybrid_clock.cc:648] HybridClock initialized: now 1781924770610500 us; error 45 us; skew 500 ppm
I20260620 03:06:10.610865 6346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:10 dist-test-slave-xv68 krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:10.613989 6346 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:06:10.615590 6346 webserver.cc:460] Webserver started at http://127.0.97.3:35267/ using document root <none> and password file <none>
I20260620 03:06:10.616307 6346 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:10.616464 6346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:10.616775 6346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:10.620009 6346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "4db20647cdf648c989bd7958de51af14"
format_stamp: "Formatted at 2026-06-20 03:06:10 on dist-test-slave-xv68"
server_key: "54a03d280cadd75fffba389165d0d7f1"
server_key_iv: "13794fac2b3d484eb24372520fc36527"
server_key_version: "encryptionkey@0"
I20260620 03:06:10.620846 6346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4db20647cdf648c989bd7958de51af14"
format_stamp: "Formatted at 2026-06-20 03:06:10 on dist-test-slave-xv68"
server_key: "54a03d280cadd75fffba389165d0d7f1"
server_key_iv: "13794fac2b3d484eb24372520fc36527"
server_key_version: "encryptionkey@0"
I20260620 03:06:10.626794 6346 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260620 03:06:10.630708 6361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:10.632179 6346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:06:10.632390 6346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "4db20647cdf648c989bd7958de51af14"
format_stamp: "Formatted at 2026-06-20 03:06:10 on dist-test-slave-xv68"
server_key: "54a03d280cadd75fffba389165d0d7f1"
server_key_iv: "13794fac2b3d484eb24372520fc36527"
server_key_version: "encryptionkey@0"
I20260620 03:06:10.632618 6346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:10.645480 6346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:10.649255 6346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:10.649531 6346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:10.650424 6346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:10.651751 6346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:10.651855 6346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:10.651991 6346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:10.652062 6346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:10.684371 6346 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39729
I20260620 03:06:10.684413 6473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39729 every 8 connection(s)
I20260620 03:06:10.685913 6346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:06:10.696084 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 6346
I20260620 03:06:10.696259 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:06:10.696800 388 external_mini_cluster.cc:1428] Setting key 7e8a17022687fd75d59012bb4ffafddb
Jun 20 03:06:10 dist-test-slave-xv68 krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:06:10.705775 6209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:10.688586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40367 (local address 127.0.97.62:34615)
0620 03:06:10.688767 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:10.688771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:10.690039 (+ 1268us) server_negotiation.cc:407] Connection header received
0620 03:06:10.691873 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:10.691877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:10.691940 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:10.692026 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:10.693500 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:10.694270 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:10.695919 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:10.696586 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:10.696717 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:10.700630 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:10.700635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:10.700638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:10.700669 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:10.703035 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:10.703780 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:10.703784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:10.703786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:10.703902 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:10.704252 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:10.704255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:10.704257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:10.704520 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:10.704623 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:10.705230 (+ 607us) server_negotiation.cc:299] Negotiation successful
0620 03:06:10.705523 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 03:06:10.706736 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:10.689105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34615 (local address 127.0.97.3:40367)
0620 03:06:10.689870 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:10.689911 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:10.691615 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:10.692228 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:10.692236 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:10.692773 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:10.693309 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:10.693325 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:10.694438 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:10.694442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:10.695751 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:10.695767 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:10.696796 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:10.696802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:10.697060 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:10.697832 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:10.697854 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:10.700431 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:10.703165 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:10.703172 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:10.703184 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:10.703674 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:10.703991 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:10.703994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:10.703996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:10.704160 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:10.704620 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:10.704625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:10.704916 (+ 291us) client_negotiation.cc:769] Sending connection context
0620 03:06:10.705134 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 03:06:10.705585 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":329,"threads_started":1}
I20260620 03:06:10.708076 6474 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34615
I20260620 03:06:10.708387 6474 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:10.709087 6474 heartbeater.cc:511] Master 127.0.97.62:34615 requested a full tablet report, sending...
I20260620 03:06:10.710547 6018 ts_manager.cc:194] Registered new tserver with Master: 4db20647cdf648c989bd7958de51af14 (127.0.97.3:39729)
I20260620 03:06:10.711659 6018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40367
I20260620 03:06:10.712419 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/test-admin.keytab.
Jun 20 03:06:10 dist-test-slave-xv68 krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:10.741201 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 03:06:10.746167 6481 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:10.743794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.0.97.1:41933)
0620 03:06:10.744447 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:10.744454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:10.744478 (+ 24us) server_negotiation.cc:407] Connection header received
0620 03:06:10.744535 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:10.744541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:10.744742 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:10.744913 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:10.745767 (+ 854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55166: BlockingRecv error: recv got EOF from 127.0.0.1:55166 (error 108)
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":305,"threads_started":1}
W20260620 03:06:10.746277 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 03:06:10 dist-test-slave-xv68 krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:10.747938 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 03:06:10 dist-test-slave-xv68 krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:10.749274 388 init.cc:295] Successfully reacquired a new kerberos TGT
I20260620 03:06:10.750362 6207 heartbeater.cc:503] Master 127.0.97.62:34615 was elected leader, sending a full tablet report...
Jun 20 03:06:10 dist-test-slave-xv68 krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:06:10.764595 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:10.750937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55182 (local address 127.0.97.1:41933)
0620 03:06:10.751079 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:10.751084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:10.751426 (+ 342us) server_negotiation.cc:407] Connection header received
0620 03:06:10.751618 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:10.751623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:10.751691 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:10.751799 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:10.753014 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:10.753950 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:10.755162 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:10.755766 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:10.755869 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:10.759333 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:10.759350 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:10.759365 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:10.759400 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:10.762147 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:10.762784 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:10.762791 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:10.762793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:10.762906 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:10.763229 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:10.763232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:10.763234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:10.763632 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:10.763831 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:10.764153 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 03:06:10.764332 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 03:06:10.770234 6481 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:10.768418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.0.97.1:41933)
0620 03:06:10.768614 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:10.768622 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:10.768914 (+ 292us) server_negotiation.cc:407] Connection header received
0620 03:06:10.769086 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:10.769097 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:10.769149 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:10.769246 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:10.769967 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55198: BlockingRecv error: recv got EOF from 127.0.0.1:55198 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 03:06:10.770709 388 init.cc:244] Error while opening credential cache '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/krb5cc)
Jun 20 03:06:10 dist-test-slave-xv68 krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:10.772217 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 03:06:10 dist-test-slave-xv68 krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:10.773586 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 03:06:10 dist-test-slave-xv68 krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:06:10.787585 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:10.774804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55202 (local address 127.0.97.1:41933)
0620 03:06:10.775004 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:10.775012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:10.775258 (+ 246us) server_negotiation.cc:407] Connection header received
0620 03:06:10.775495 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:10.775499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:10.775550 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:10.775631 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:10.776577 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:10.777254 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:10.778453 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:10.778870 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:10.778979 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:10.782438 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:10.782443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:10.782446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:10.782475 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:10.785156 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:10.785771 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:10.785775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:10.785777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:10.785892 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:10.786321 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:10.786324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:10.786327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:10.786653 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:10.786840 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:10.787076 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 03:06:10.787218 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 03:06:10.791525 6481 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:10.789958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.0.97.1:41933)
0620 03:06:10.790151 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:10.790160 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:10.790436 (+ 276us) server_negotiation.cc:407] Connection header received
0620 03:06:10.790560 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:10.790563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:10.790623 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:10.790695 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:10.791278 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55212: BlockingRecv error: recv got EOF from 127.0.0.1:55212 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 03:06:10.791697 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924720365844-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 03:06:10 dist-test-slave-xv68 krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:10.793229 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 03:06:10 dist-test-slave-xv68 krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:10.794432 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 03:06:10 dist-test-slave-xv68 krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:06:10.808080 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:10.795700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.0.97.1:41933)
0620 03:06:10.795898 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:10.795905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:10.796145 (+ 240us) server_negotiation.cc:407] Connection header received
0620 03:06:10.796320 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:10.796324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:10.796411 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:10.796514 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:10.797442 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:10.798140 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:10.799404 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:10.799822 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:10.799954 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:10.803415 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:10.803419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:10.803422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:10.803450 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:10.805854 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:10.806462 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:10.806466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:10.806468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:10.806597 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:10.806902 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:10.806906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:10.806908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:10.807149 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:10.807456 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:10.807670 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 03:06:10.807821 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:06:10.809556 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 6079
I20260620 03:06:10.820990 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:10.810376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55222 (local address 127.0.97.1:41933)
0620 03:06:10.810522 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:10.810526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:10.810826 (+ 300us) server_negotiation.cc:407] Connection header received
0620 03:06:10.810946 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:10.810950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:10.810998 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:10.811072 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:10.812016 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:10.812710 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:10.813863 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:10.814268 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:10.814368 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:10.816355 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:10.816359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:10.816362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:10.816407 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:10.818698 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:10.819274 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:10.819277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:10.819279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:10.819492 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:10.819813 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:10.819816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:10.819818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:10.820055 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:10.820240 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:10.820594 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 03:06:10.820743 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 03:06:10.821648 6202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:10.906119 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 6079
I20260620 03:06:10.922224 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 6213
Jun 20 03:06:10 dist-test-slave-xv68 krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:06:10.937731 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:10.923115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.0.97.2:44575)
0620 03:06:10.923675 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:10.923682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:10.923707 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:06:10.923765 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:10.923771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:10.923985 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:10.924115 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:10.925126 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:10.926179 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:10.927524 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:10.928094 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:10.928226 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:10.932079 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:10.932092 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:10.932108 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:10.932144 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:10.934867 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:10.935684 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:10.935691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:10.935693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:10.935817 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:10.936214 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:10.936218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:10.936220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:10.936635 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:10.936868 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:10.937220 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 03:06:10.937450 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":306,"threads_started":1}
I20260620 03:06:10.938935 6336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:11.020334 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 6213
I20260620 03:06:11.036408 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 6346
Jun 20 03:06:11 dist-test-slave-xv68 krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:06:11.052918 6496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:11.037500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33924 (local address 127.0.97.3:39729)
0620 03:06:11.038136 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:11.038143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:11.038179 (+ 36us) server_negotiation.cc:407] Connection header received
0620 03:06:11.038240 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:11.038246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:11.038472 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:11.038654 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:11.039922 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:11.041026 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:11.042481 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:11.043107 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:11.043269 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:11.047466 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:11.047477 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:11.047492 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:11.047521 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:11.050171 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:11.050845 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:11.050860 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:11.050862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:11.050974 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:11.051477 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:11.051484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:11.051488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:11.051809 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:11.052130 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:11.052404 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 03:06:11.052633 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":320,"threads_started":1}
I20260620 03:06:11.054132 6469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:11.142627 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 6346
I20260620 03:06:11.157691 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 5986
I20260620 03:06:11.158818 6048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:11.259938 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 5986
2026-06-20T03:06:11Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3858 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:06:11 dist-test-slave-xv68 krb5kdc[6501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:06:11 dist-test-slave-xv68 krb5kdc[6501](info): set up 2 sockets
Jun 20 03:06:11 dist-test-slave-xv68 krb5kdc[6501](info): commencing operation
krb5kdc: starting...
W20260620 03:06:12.372045 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:06:12 dist-test-slave-xv68 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:06:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:06:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 03:06:12.574270 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45499
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:12.890707 6514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:12.890980 6514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:12.891130 6514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:12.898229 6514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:06:12.898326 6514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:12.898427 6514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:12.898500 6514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:06:12.898566 6514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:06:12.909546 6514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:12.911031 6514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:12.912390 6514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:12.924057 6519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:12.924104 6520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:12.924146 6522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:12.925014 6514 server_base.cc:1034] running on GCE node
I20260620 03:06:12.925792 6514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:12.927162 6514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:12.928400 6514 hybrid_clock.cc:648] HybridClock initialized: now 1781924772928339 us; error 80 us; skew 500 ppm
I20260620 03:06:12.928790 6514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:12 dist-test-slave-xv68 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924772, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:12.931851 6514 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 03:06:12.933352 6514 webserver.cc:460] Webserver started at http://127.0.97.62:32905/ using document root <none> and password file <none>
I20260620 03:06:12.933992 6514 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:12.934090 6514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:12.934332 6514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:12.937234 6514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "85a88abe479747acb7a7878d64c9d24c"
format_stamp: "Formatted at 2026-06-20 03:06:12 on dist-test-slave-xv68"
server_key: "ee296202e8e33ee21c88bb496cf08307"
server_key_iv: "289d2e81f662d21404cdc8764b91ba74"
server_key_version: "encryptionkey@0"
I20260620 03:06:12.937917 6514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "85a88abe479747acb7a7878d64c9d24c"
format_stamp: "Formatted at 2026-06-20 03:06:12 on dist-test-slave-xv68"
server_key: "ee296202e8e33ee21c88bb496cf08307"
server_key_iv: "289d2e81f662d21404cdc8764b91ba74"
server_key_version: "encryptionkey@0"
I20260620 03:06:12.942862 6514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:06:12.946568 6529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:12.948144 6514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 03:06:12.948329 6514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "85a88abe479747acb7a7878d64c9d24c"
format_stamp: "Formatted at 2026-06-20 03:06:12 on dist-test-slave-xv68"
server_key: "ee296202e8e33ee21c88bb496cf08307"
server_key_iv: "289d2e81f662d21404cdc8764b91ba74"
server_key_version: "encryptionkey@0"
I20260620 03:06:12.948515 6514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:12.977557 6514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:12.981657 6514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:12.981949 6514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:13.000531 6514 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45499
I20260620 03:06:13.000572 6580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45499 every 8 connection(s)
I20260620 03:06:13.001878 6514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:06:13.002517 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 6514
I20260620 03:06:13.002736 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:06:13.003563 388 external_mini_cluster.cc:1428] Setting key c4034828c2c914c836a2916346daa92d
I20260620 03:06:13.006220 6581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 03:06:13 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 03:06:13.018301 6581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c: Bootstrap starting.
I20260620 03:06:13.021699 6581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:13.022698 6581 log.cc:824] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:13.026166 6584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:13.005508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.0.97.62:45499)
0620 03:06:13.006247 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:13.006259 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:13.006297 (+ 38us) server_negotiation.cc:407] Connection header received
0620 03:06:13.007917 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:13.007937 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:13.008286 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:13.008613 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:13.010367 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:13.012527 (+ 2160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:13.013788 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:13.014779 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:13.014977 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:13.018721 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:13.018729 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:13.018742 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:13.018775 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:13.021676 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:13.022336 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:13.022346 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:13.022357 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:13.022548 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:13.022959 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:13.022965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:13.022970 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:13.023670 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:13.023891 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:13.024461 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 03:06:13.024863 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":344,"threads_started":1}
I20260620 03:06:13.026208 6581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c: No bootstrap required, opened a new log
I20260620 03:06:13.034193 6581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260620 03:06:13.034515 6581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:13.034584 6581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a88abe479747acb7a7878d64c9d24c, State: Initialized, Role: FOLLOWER
I20260620 03:06:13.035130 6581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260620 03:06:13.035391 6581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:06:13.035495 6581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:06:13.035638 6581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:13.037629 6581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260620 03:06:13.038102 6581 leader_election.cc:302] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85a88abe479747acb7a7878d64c9d24c; no voters:
I20260620 03:06:13.038776 6581 leader_election.cc:288] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:06:13.038895 6586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:13.039552 6586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [term 1 LEADER]: Becoming Leader. State: Replica: 85a88abe479747acb7a7878d64c9d24c, State: Running, Role: LEADER
I20260620 03:06:13.040124 6586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260620 03:06:13.040954 6581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:06:13.042936 6587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } }
I20260620 03:06:13.043310 6587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:13.043933 6588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85a88abe479747acb7a7878d64c9d24c. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } }
I20260620 03:06:13.044345 6588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:13.045917 6594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:06:13.051158 6594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:06:13.059250 6594 catalog_manager.cc:1347] Generated new cluster ID: eec7f634163546948ee74189cc31cdbb
I20260620 03:06:13.059442 6594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:06:13.087589 6594 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:06:13.089052 6594 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:06:13.097141 6594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c: Generated new TSK 0
I20260620 03:06:13.097994 6594 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 03:06:13.155236 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45499
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:13.490432 6607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:13.490697 6607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:13.490788 6607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:13.497821 6607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:13.497954 6607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:13.498070 6607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:06:13.507470 6607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:13.508620 6607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:13.509843 6607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:13.522660 6615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:13.522689 6613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:13.522718 6612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:13.523635 6607 server_base.cc:1034] running on GCE node
I20260620 03:06:13.524207 6607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:13.525126 6607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:13.526360 6607 hybrid_clock.cc:648] HybridClock initialized: now 1781924773526303 us; error 47 us; skew 500 ppm
I20260620 03:06:13.526737 6607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:13 dist-test-slave-xv68 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924773, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:13.529847 6607 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 03:06:13.531212 6607 webserver.cc:460] Webserver started at http://127.0.97.1:39691/ using document root <none> and password file <none>
I20260620 03:06:13.531912 6607 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:13.532055 6607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:13.532352 6607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:13.535178 6607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "63870f6d79a144b7bd7696c4c4c47b4d"
format_stamp: "Formatted at 2026-06-20 03:06:13 on dist-test-slave-xv68"
server_key: "4ee44307c91558d9c3c4572f6aec2e1d"
server_key_iv: "c8e22aec13bd136b48060713f5a52b51"
server_key_version: "encryptionkey@0"
I20260620 03:06:13.536067 6607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "63870f6d79a144b7bd7696c4c4c47b4d"
format_stamp: "Formatted at 2026-06-20 03:06:13 on dist-test-slave-xv68"
server_key: "4ee44307c91558d9c3c4572f6aec2e1d"
server_key_iv: "c8e22aec13bd136b48060713f5a52b51"
server_key_version: "encryptionkey@0"
I20260620 03:06:13.540974 6607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:06:13.544559 6622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:13.546013 6607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:06:13.546182 6607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "63870f6d79a144b7bd7696c4c4c47b4d"
format_stamp: "Formatted at 2026-06-20 03:06:13 on dist-test-slave-xv68"
server_key: "4ee44307c91558d9c3c4572f6aec2e1d"
server_key_iv: "c8e22aec13bd136b48060713f5a52b51"
server_key_version: "encryptionkey@0"
I20260620 03:06:13.546396 6607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:13.559566 6607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:13.563378 6607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:13.563690 6607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:13.564605 6607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:13.565877 6607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:13.565985 6607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:13.566129 6607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:13.566206 6607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:13.598999 6607 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38365
I20260620 03:06:13.599047 6734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38365 every 8 connection(s)
I20260620 03:06:13.600301 6607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:06:13.607368 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 6607
I20260620 03:06:13.607592 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:06:13.608208 388 external_mini_cluster.cc:1428] Setting key 64ce692de33f72f3e9ee7d0540c60437
Jun 20 03:06:13 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924773, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 03:06:13.620956 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:13.603037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33971 (local address 127.0.97.62:45499)
0620 03:06:13.603525 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:13.603529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:13.604408 (+ 879us) server_negotiation.cc:407] Connection header received
0620 03:06:13.606227 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:13.606232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:13.606315 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:13.606431 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:13.608076 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:13.608901 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:13.610857 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:13.611287 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:13.611458 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:13.615484 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:13.615489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:13.615492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:13.615526 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:13.617950 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:13.618683 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:13.618695 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:13.618697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:13.618809 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:13.619235 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:13.619239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:13.619242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:13.619549 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:13.619711 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:13.620368 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 03:06:13.620581 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":278,"threads_started":1}
I20260620 03:06:13.621920 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:13.603555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:33971)
0620 03:06:13.604249 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:13.604296 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:13.605936 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:13.606669 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:13.606677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:13.607199 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:13.607906 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:13.607925 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:13.609072 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:13.609076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:13.610710 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:13.610722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:13.611563 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:13.611574 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:13.611894 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:13.612614 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:13.612646 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:13.615202 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:13.618111 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:13.618120 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:13.618136 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:13.618555 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:13.618918 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:13.618924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:13.618928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:13.619117 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:13.619675 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:13.619683 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:13.620015 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 03:06:13.620267 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 03:06:13.620817 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":272,"threads_started":1}
I20260620 03:06:13.623562 6735 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45499
I20260620 03:06:13.623906 6735 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:13.624637 6735 heartbeater.cc:511] Master 127.0.97.62:45499 requested a full tablet report, sending...
I20260620 03:06:13.626688 6546 ts_manager.cc:194] Registered new tserver with Master: 63870f6d79a144b7bd7696c4c4c47b4d (127.0.97.1:38365)
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 03:06:13.628213 6546 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 03:06:13.652889 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45499
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:13.978098 6741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:13.978431 6741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:13.978613 6741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:13.985777 6741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:13.985952 6741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:13.986155 6741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:06:13.995900 6741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:13.997256 6741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:13.998615 6741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:14.011833 6747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:14.011861 6746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:14.012931 6749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:14.013641 6741 server_base.cc:1034] running on GCE node
I20260620 03:06:14.014288 6741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:14.015224 6741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:14.016454 6741 hybrid_clock.cc:648] HybridClock initialized: now 1781924774016407 us; error 43 us; skew 500 ppm
I20260620 03:06:14.016775 6741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:14 dist-test-slave-xv68 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924774, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:14.019841 6741 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 03:06:14.021175 6741 webserver.cc:460] Webserver started at http://127.0.97.2:41637/ using document root <none> and password file <none>
I20260620 03:06:14.021883 6741 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:14.021982 6741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:14.022210 6741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:14.025033 6741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "b56f067b01d34ef1b8c339beaf862a84"
format_stamp: "Formatted at 2026-06-20 03:06:14 on dist-test-slave-xv68"
server_key: "dcdaaa89b00cef25616b0084fc08c69e"
server_key_iv: "cf2af04c8996eae1794da1d2912f2803"
server_key_version: "encryptionkey@0"
I20260620 03:06:14.025753 6741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b56f067b01d34ef1b8c339beaf862a84"
format_stamp: "Formatted at 2026-06-20 03:06:14 on dist-test-slave-xv68"
server_key: "dcdaaa89b00cef25616b0084fc08c69e"
server_key_iv: "cf2af04c8996eae1794da1d2912f2803"
server_key_version: "encryptionkey@0"
I20260620 03:06:14.030910 6741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 03:06:14.034739 6756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:14.036142 6741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:06:14.036309 6741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "b56f067b01d34ef1b8c339beaf862a84"
format_stamp: "Formatted at 2026-06-20 03:06:14 on dist-test-slave-xv68"
server_key: "dcdaaa89b00cef25616b0084fc08c69e"
server_key_iv: "cf2af04c8996eae1794da1d2912f2803"
server_key_version: "encryptionkey@0"
I20260620 03:06:14.036495 6741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:14.068606 6741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:14.072820 6741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:14.073110 6741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:14.074096 6741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:14.075467 6741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:14.075541 6741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:14.075664 6741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:14.075722 6741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:14.108810 6741 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35015
I20260620 03:06:14.108898 6868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35015 every 8 connection(s)
I20260620 03:06:14.110085 6741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:06:14.113164 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 6741
I20260620 03:06:14.113317 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:06:14.113760 388 external_mini_cluster.cc:1428] Setting key f6f080a39a26c50f4b412aaed622ecb4
Jun 20 03:06:14 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924774, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 03:06:14.131325 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:14.112722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44345 (local address 127.0.97.62:45499)
0620 03:06:14.112951 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:14.112958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:14.114301 (+ 1343us) server_negotiation.cc:407] Connection header received
0620 03:06:14.116419 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:14.116435 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:14.116520 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:14.116626 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:14.118072 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:14.119174 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:14.121084 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:14.121747 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:14.121846 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:14.125613 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:14.125618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:14.125622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:14.125653 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:14.128274 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:14.129012 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:14.129022 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:14.129025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:14.129144 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:14.129588 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:14.129597 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:14.129599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:14.129844 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:14.130020 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:14.130725 (+ 705us) server_negotiation.cc:299] Negotiation successful
0620 03:06:14.130955 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 03:06:14.132411 6871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:14.113326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:44345)
0620 03:06:14.114147 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:14.114197 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:14.116157 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:14.116824 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:14.116833 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:14.117308 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:14.117902 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:14.117918 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:14.119470 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:14.119475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:14.120933 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:14.120946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:14.122001 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:14.122009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:14.122369 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:14.123147 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:14.123170 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:14.125378 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:14.128432 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:14.128442 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:14.128462 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:14.128877 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:14.129256 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:14.129263 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:14.129266 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:14.129465 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:14.129982 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:14.129990 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:14.130351 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 03:06:14.130638 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 03:06:14.131181 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":352,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 03:06:14.134013 6869 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45499
I20260620 03:06:14.134346 6869 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:14.135072 6869 heartbeater.cc:511] Master 127.0.97.62:45499 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260620 03:06:14.136782 6546 ts_manager.cc:194] Registered new tserver with Master: b56f067b01d34ef1b8c339beaf862a84 (127.0.97.2:35015)
I20260620 03:06:14.137573 6546 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 03:06:14.162170 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45499
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:14.483093 6874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:14.483371 6874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:14.483534 6874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:14.490618 6874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:14.490722 6874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:14.490895 6874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:06:14.500502 6874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:14.501770 6874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:14.503095 6874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:14.516067 6879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:14.516091 6880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:14.516223 6882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:14.517148 6874 server_base.cc:1034] running on GCE node
I20260620 03:06:14.517786 6874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:14.518682 6874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:14.519870 6874 hybrid_clock.cc:648] HybridClock initialized: now 1781924774519827 us; error 38 us; skew 500 ppm
I20260620 03:06:14.520179 6874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:14 dist-test-slave-xv68 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924774, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:14.523169 6874 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 03:06:14.524518 6874 webserver.cc:460] Webserver started at http://127.0.97.3:46791/ using document root <none> and password file <none>
I20260620 03:06:14.525166 6874 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:14.525309 6874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:14.525627 6874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:14.528474 6874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853"
format_stamp: "Formatted at 2026-06-20 03:06:14 on dist-test-slave-xv68"
server_key: "cba4bb4416eb6048f720d884a2d1ec37"
server_key_iv: "fec4a2794fdd0c86ba9c4dc6ee384df1"
server_key_version: "encryptionkey@0"
I20260620 03:06:14.529248 6874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853"
format_stamp: "Formatted at 2026-06-20 03:06:14 on dist-test-slave-xv68"
server_key: "cba4bb4416eb6048f720d884a2d1ec37"
server_key_iv: "fec4a2794fdd0c86ba9c4dc6ee384df1"
server_key_version: "encryptionkey@0"
I20260620 03:06:14.534193 6874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:06:14.537581 6889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:14.538864 6874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 03:06:14.539039 6874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853"
format_stamp: "Formatted at 2026-06-20 03:06:14 on dist-test-slave-xv68"
server_key: "cba4bb4416eb6048f720d884a2d1ec37"
server_key_iv: "fec4a2794fdd0c86ba9c4dc6ee384df1"
server_key_version: "encryptionkey@0"
I20260620 03:06:14.539235 6874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:14.561192 6874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:14.565169 6874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:14.565469 6874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:14.566403 6874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:14.567720 6874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:14.567795 6874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:14.567966 6874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:14.568018 6874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:14.601215 6874 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35005
I20260620 03:06:14.601257 7001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35005 every 8 connection(s)
I20260620 03:06:14.602770 6874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:06:14.612064 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 6874
I20260620 03:06:14.612279 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:06:14.612805 388 external_mini_cluster.cc:1428] Setting key e18e916e3cc14a62dd0af2ae88fbc61d
Jun 20 03:06:14 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924774, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 03:06:14.622277 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:14.605448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37171 (local address 127.0.97.62:45499)
0620 03:06:14.605618 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:14.605623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:14.606861 (+ 1238us) server_negotiation.cc:407] Connection header received
0620 03:06:14.608665 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:14.608670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:14.608733 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:14.608819 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:14.610161 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:14.610917 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:14.612665 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:14.613277 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:14.613424 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:14.617326 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:14.617330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:14.617333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:14.617363 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:14.619747 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:14.620415 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:14.620419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:14.620421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:14.620553 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:14.620914 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:14.620917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:14.620919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:14.621144 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:14.621248 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:14.621899 (+ 651us) server_negotiation.cc:299] Negotiation successful
0620 03:06:14.622052 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 03:06:14.623202 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:14.605920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:37171)
0620 03:06:14.606651 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:14.606722 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:14.608405 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:14.609023 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:14.609031 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:14.609497 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:14.610021 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:14.610037 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:14.611039 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:14.611043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:14.612522 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:14.612534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:14.613603 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:14.613613 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:14.613901 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:14.614733 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:14.614771 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:14.617138 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:14.619865 (+ 2727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:14.619872 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:14.619882 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:14.620310 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:14.620644 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:14.620647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:14.620650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:14.620818 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:14.621249 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:14.621254 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:14.621533 (+ 279us) client_negotiation.cc:769] Sending connection context
0620 03:06:14.621752 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 03:06:14.622132 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":268,"threads_started":1}
I20260620 03:06:14.624560 7002 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45499
I20260620 03:06:14.624833 7002 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:14.625475 7002 heartbeater.cc:511] Master 127.0.97.62:45499 requested a full tablet report, sending...
I20260620 03:06:14.626806 6546 ts_manager.cc:194] Registered new tserver with Master: c4a1d8ef8d9b4cc391b7a657fb81a853 (127.0.97.3:35005)
I20260620 03:06:14.627774 6546 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37171
I20260620 03:06:14.628224 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:06:14.631598 6735 heartbeater.cc:503] Master 127.0.97.62:45499 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260620 03:06:14.674445 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44195
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45499,127.0.97.61:44195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:14.986887 7007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:14.987155 7007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:14.987318 7007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:14.994491 7007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:06:14.994634 7007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:14.994742 7007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:14.994844 7007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:06:14.994930 7007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:06:15.004729 7007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45499,127.0.97.61:44195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
W20260620 03:06:15.005107 7007 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:45499,127.0.97.61:44195'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 03:06:15.006124 7007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:15.007484 7007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:15.019055 7012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:15.019104 7013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:15.019006 7015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:15.020243 7007 server_base.cc:1034] running on GCE node
I20260620 03:06:15.020998 7007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:15.022269 7007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:15.023510 7007 hybrid_clock.cc:648] HybridClock initialized: now 1781924775023418 us; error 76 us; skew 500 ppm
I20260620 03:06:15.023806 7007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:15 dist-test-slave-xv68 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924775, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:15.030380 7007 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 03:06:15.032008 7007 webserver.cc:460] Webserver started at http://127.0.97.61:35189/ using document root <none> and password file <none>
I20260620 03:06:15.032703 7007 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:15.032788 7007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:15.033044 7007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:15.035975 7007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/data/instance:
uuid: "5f373e2277904013a4c4acee64771191"
format_stamp: "Formatted at 2026-06-20 03:06:15 on dist-test-slave-xv68"
server_key: "83114c564a11e51744731d687d0081b1"
server_key_iv: "4a09c79fcd26e1143cb994e05af232e3"
server_key_version: "encryptionkey@0"
I20260620 03:06:15.036684 7007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/wal/instance:
uuid: "5f373e2277904013a4c4acee64771191"
format_stamp: "Formatted at 2026-06-20 03:06:15 on dist-test-slave-xv68"
server_key: "83114c564a11e51744731d687d0081b1"
server_key_iv: "4a09c79fcd26e1143cb994e05af232e3"
server_key_version: "encryptionkey@0"
I20260620 03:06:15.041858 7007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 03:06:15.045367 7022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:15.046623 7007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:06:15.046794 7007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/wal
uuid: "5f373e2277904013a4c4acee64771191"
format_stamp: "Formatted at 2026-06-20 03:06:15 on dist-test-slave-xv68"
server_key: "83114c564a11e51744731d687d0081b1"
server_key_iv: "4a09c79fcd26e1143cb994e05af232e3"
server_key_version: "encryptionkey@0"
I20260620 03:06:15.046996 7007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:15.060570 7007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:15.064340 7007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:15.064597 7007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:15.082229 7007 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44195
I20260620 03:06:15.082252 7073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44195 every 8 connection(s)
I20260620 03:06:15.083580 7007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/data/info.pb
I20260620 03:06:15.087373 7074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:15.092172 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 7007
I20260620 03:06:15.095492 7074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:06:15 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 03:06:15 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924775, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 03:06:15.124480 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.101767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39886 (local address 127.0.97.62:45499)
0620 03:06:15.102298 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:15.102306 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:15.102801 (+ 495us) server_negotiation.cc:407] Connection header received
0620 03:06:15.103052 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:15.103060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:15.103146 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:15.103268 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:15.105387 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.106618 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.109000 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.109670 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.109792 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.115552 (+ 5760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:15.115559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:15.115564 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:15.115643 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:15.120034 (+ 4391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.121202 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.121210 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.121214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.121397 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.121969 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.121974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.121978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.122460 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:15.122597 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:15.123737 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0620 03:06:15.124030 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 03:06:15.126195 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.094868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49484 (local address 127.0.97.61:44195)
0620 03:06:15.095885 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:15.095903 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:15.095966 (+ 63us) server_negotiation.cc:407] Connection header received
0620 03:06:15.099491 (+ 3525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:15.099526 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:15.100072 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:15.100761 (+ 689us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:15.102641 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.105273 (+ 2632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.107213 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.108420 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.108598 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.114714 (+ 6116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:15.114725 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:15.114741 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:15.114784 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:15.121558 (+ 6774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.122509 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.122521 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.122528 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.122688 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.123241 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.123246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.123250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.123625 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:15.123941 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:15.124510 (+ 569us) server_negotiation.cc:299] Negotiation successful
0620 03:06:15.124946 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":783,"thread_start_us":451,"threads_started":1}
I20260620 03:06:15.126283 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.101791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:39886)
0620 03:06:15.102675 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:15.102715 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:15.102891 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:15.103350 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:15.103357 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:15.104305 (+ 948us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:15.105162 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.105188 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.106805 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.106811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.108828 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.108848 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.110126 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.110133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.110347 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.111372 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:15.111493 (+ 121us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:15.115258 (+ 3765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:15.120217 (+ 4959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.120231 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.120243 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.121040 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.121558 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.121564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.121567 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.121815 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.122688 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:15.122697 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:15.123163 (+ 466us) client_negotiation.cc:769] Sending connection context
0620 03:06:15.124742 (+ 1579us) client_negotiation.cc:240] Negotiation successful
0620 03:06:15.125112 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":60,"thread_start_us":482,"threads_started":1}
I20260620 03:06:15.128515 7074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:06:15.140398 6869 heartbeater.cc:503] Master 127.0.97.62:45499 was elected leader, sending a full tablet report...
Jun 20 03:06:15 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924775, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 03:06:15.148193 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.129216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44195 (local address 127.0.0.1:49488)
0620 03:06:15.129419 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:15.129438 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:15.129743 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:15.130476 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:15.130483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:15.131030 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:15.131268 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.131281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.132971 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.132979 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.134682 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.134701 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.135758 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.135768 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.136007 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.139427 (+ 3420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:15.139458 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:15.142452 (+ 2994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:15.145837 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.145843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.145847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.146217 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.146570 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.146577 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.146583 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.146707 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.147205 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:15.147210 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:15.147399 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 03:06:15.147537 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 03:06:15.147822 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 03:06:15.148231 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.129702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.0.97.61:44195)
0620 03:06:15.130137 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:15.130142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:15.130156 (+ 14us) server_negotiation.cc:407] Connection header received
0620 03:06:15.130217 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:15.130223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:15.130285 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:15.130415 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:15.131617 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.132804 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.134855 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.135576 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.135718 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.142631 (+ 6913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:15.142639 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:15.142647 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:15.142684 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:15.145683 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.146328 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.146335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.146338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.146460 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.146812 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.146817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.146820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.147097 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:15.147248 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:15.147609 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 03:06:15.147942 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":153,"server-negotiator.queue_time_us":80}
I20260620 03:06:15.153347 7074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: Bootstrap starting.
I20260620 03:06:15.156833 7074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:15.157853 7074 log.cc:824] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:15.160737 7074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: No bootstrap required, opened a new log
I20260620 03:06:15.167495 7074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } }
I20260620 03:06:15.167827 7074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:15.167927 7074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f373e2277904013a4c4acee64771191, State: Initialized, Role: FOLLOWER
I20260620 03:06:15.168507 7074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } }
I20260620 03:06:15.169334 7080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } } }
I20260620 03:06:15.169689 7080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:06:15.170418 7074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 03:06:15.179168 7091 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 03:06:15.179404 7091 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 03:06:15.181851 7007 master_runner.cc:428] Detected that this master 5f373e2277904013a4c4acee64771191 is joining an existing cluster
I20260620 03:06:15.181953 7007 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:44195
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
I20260620 03:06:15.219446 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.193608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39900 (local address 127.0.97.62:45499)
0620 03:06:15.193802 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:15.193811 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:15.203857 (+ 10046us) server_negotiation.cc:407] Connection header received
0620 03:06:15.204206 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:15.204214 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:15.204299 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:15.204375 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:15.206091 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.207215 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.209326 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.209961 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.210074 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.212646 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:15.212652 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:15.212656 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:15.212692 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:15.216270 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.217048 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.217054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.217057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.217203 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.217703 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.217711 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.217715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.218109 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:15.218232 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:15.218813 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 03:06:15.219030 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 03:06:15.219521 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.195027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49504 (local address 127.0.97.61:44195)
0620 03:06:15.195226 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:15.195235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:15.204218 (+ 8983us) server_negotiation.cc:407] Connection header received
0620 03:06:15.204384 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:15.204388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:15.204442 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:15.204516 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:15.206223 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.207438 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.208962 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.209521 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.209637 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.212635 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:15.212640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:15.212646 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:15.212681 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:15.216702 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.217499 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.217503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.217507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.217630 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.218008 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.218013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.218015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.218376 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:15.218485 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:15.218944 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 03:06:15.219141 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 03:06:15.220569 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.194888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44195 (local address 127.0.0.1:49504)
0620 03:06:15.204139 (+ 9251us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:15.204157 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:15.204280 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:15.204681 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:15.204684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:15.205569 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:15.206046 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.206063 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.207628 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.207631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.208808 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.208822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.209888 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.209895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.210070 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.210789 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:15.211106 (+ 317us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:15.212441 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:15.216837 (+ 4396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.216845 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.216848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.217387 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.217751 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.217756 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.217762 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.217891 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.218563 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:15.218567 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:15.218700 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 03:06:15.219713 (+ 1013us) client_negotiation.cc:240] Negotiation successful
0620 03:06:15.220044 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9109,"mutex_wait_us":25,"thread_start_us":8936,"threads_started":1}
I20260620 03:06:15.220649 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.194060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:39900)
0620 03:06:15.203692 (+ 9632us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:15.203716 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:15.204015 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:15.204590 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:15.204595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:15.205399 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:15.205885 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.205904 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.207414 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.207421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.209175 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.209189 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.210313 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.210322 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.210551 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.211172 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:15.211200 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:15.212441 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:15.216396 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.216402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.216405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.216911 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.217353 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.217357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.217360 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.217578 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.218313 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:15.218319 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:15.218510 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 03:06:15.219713 (+ 1203us) client_negotiation.cc:240] Negotiation successful
0620 03:06:15.220188 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9463,"mutex_wait_us":255,"thread_start_us":9291,"threads_started":1}
Jun 20 03:06:15 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924772, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 03:06:15.241619 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.227258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49508 (local address 127.0.97.61:44195)
0620 03:06:15.227668 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:15.227675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:15.228049 (+ 374us) server_negotiation.cc:407] Connection header received
0620 03:06:15.228399 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:15.228404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:15.228500 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:15.228606 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:15.229671 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.230408 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.231756 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.232275 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.232402 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.235681 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:15.235687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:15.235691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:15.235730 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:15.238617 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.239375 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.239380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.239382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.239579 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.239996 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.240000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.240004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.240365 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:15.240525 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:15.240979 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 03:06:15.241229 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260620 03:06:15.241698 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.227352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44195 (local address 127.0.0.1:49508)
0620 03:06:15.227937 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:15.227974 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:15.228180 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:15.228657 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:15.228664 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:15.229318 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:15.229532 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.229541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.230543 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.230548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.231626 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.231637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.232454 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.232460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.232652 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.233293 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:15.233310 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:15.235492 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:15.238773 (+ 3281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.238784 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.238792 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.239210 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.239704 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.239710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.239714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.239874 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.240506 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:15.240515 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:15.240703 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 03:06:15.240881 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 03:06:15.241209 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":323,"threads_started":1}
I20260620 03:06:15.243594 6546 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:44195: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5f373e2277904013a4c4acee64771191"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 44195
}
attrs {
promote: true
}
}
dest_uuid: "85a88abe479747acb7a7878d64c9d24c"
cas_config_opid_index: -1
I20260620 03:06:15.244550 6546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260620 03:06:15.247565 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45945
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45499,127.0.97.61:44195,127.0.97.60:45945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:06:15.252676 7049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:15.257886 7049 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 1 FOLLOWER]: Refusing update from remote peer 85a88abe479747acb7a7878d64c9d24c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 03:06:15.258355 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5f373e2277904013a4c4acee64771191 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: true } } }
I20260620 03:06:15.259759 7103 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:15.262674 7104 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:44195
I20260620 03:06:15.263080 7104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: true } } }
I20260620 03:06:15.263284 7104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:15.263926 7007 master.cc:537] Master@127.0.97.61:44195 shutting down...
W20260620 03:06:15.316455 7028 tablet.cc:2369] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:06:15.322820 7049 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5f373e2277904013a4c4acee64771191 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: true } } }
I20260620 03:06:15.326584 7112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: true } } }
I20260620 03:06:15.326910 7080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85a88abe479747acb7a7878d64c9d24c. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: true } } }
I20260620 03:06:15.327193 7080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: This master's current role is: LEARNER
I20260620 03:06:15.327212 7109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: true } } }
I20260620 03:06:15.327656 7109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: This master's current role is: LEARNER
I20260620 03:06:15.326906 7112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: This master's current role is: LEARNER
I20260620 03:06:15.330134 7007 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 1 LEARNER]: Raft consensus shutting down.
I20260620 03:06:15.330534 7007 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 1 LEARNER]: Raft consensus is shut down!
I20260620 03:06:15.330704 7007 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: stopping tablet replica
I20260620 03:06:15.337509 7007 master.cc:552] Master@127.0.97.61:44195 shutdown complete.
I20260620 03:06:15.337599 7007 master_runner.cc:305] Clearing existing system tablet
I20260620 03:06:15.338042 7007 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:15.338151 7007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:15.345010 7007 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.008s
I20260620 03:06:15.347625 7114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:15.347985 7007 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:15.348238 7007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 03:06:15.348320 7007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/wal
uuid: "5f373e2277904013a4c4acee64771191"
format_stamp: "Formatted at 2026-06-20 03:06:15 on dist-test-slave-xv68"
server_key: "83114c564a11e51744731d687d0081b1"
server_key_iv: "4a09c79fcd26e1143cb994e05af232e3"
server_key_version: "encryptionkey@0"
I20260620 03:06:15.352671 7007 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:15.359640 7007 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 03:06:15.359800 7007 log.cc:1196] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 03:06:15.360271 7007 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: Deleting consensus metadata
I20260620 03:06:15.360522 7007 master_runner.cc:315] Copying system tablet from 127.0.97.62:45499
I20260620 03:06:15.364876 7007 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:45499
I20260620 03:06:15.380427 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.365569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:39908)
0620 03:06:15.366261 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:15.366288 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:15.366440 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:15.366839 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:15.366846 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:15.367392 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:15.367672 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.367684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.369123 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.369130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.370933 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.370950 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.371944 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.371949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.372170 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.372898 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:15.372919 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:15.373974 (+ 1055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:15.377655 (+ 3681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.377660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.377664 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.378140 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.378510 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.378519 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.378522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.378691 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.379349 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:15.379356 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:15.379524 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 03:06:15.379685 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 03:06:15.379964 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":389,"threads_started":1}
I20260620 03:06:15.380429 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.365626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.0.97.62:45499)
0620 03:06:15.365814 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:15.365820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:15.366395 (+ 575us) server_negotiation.cc:407] Connection header received
0620 03:06:15.366574 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:15.366580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:15.366704 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:15.366809 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:15.367826 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.368939 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.371095 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.371782 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.371912 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.374163 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:15.374169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:15.374174 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:15.374214 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:15.377523 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.378252 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.378257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.378261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.378394 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.378810 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.378815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.378819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.379193 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:15.379362 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:15.379859 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 03:06:15.380087 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 03:06:15.381903 6566 tablet_copy_service.cc:140] P 85a88abe479747acb7a7878d64c9d24c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5f373e2277904013a4c4acee64771191 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:39908)
I20260620 03:06:15.382198 6566 tablet_copy_service.cc:161] P 85a88abe479747acb7a7878d64c9d24c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5f373e2277904013a4c4acee64771191 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:39908: session id = 5f373e2277904013a4c4acee64771191-00000000000000000000000000000000
I20260620 03:06:15.386632 6566 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 03:06:15.388765 7007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 03:06:15.395871 7007 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: tablet copy: Starting download of 0 data blocks...
I20260620 03:06:15.396170 7007 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: tablet copy: Starting download of 1 WAL segments...
I20260620 03:06:15.399690 7007 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 03:06:15.404610 6566 tablet_copy_service.cc:342] P 85a88abe479747acb7a7878d64c9d24c: Request end of tablet copy session 5f373e2277904013a4c4acee64771191-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:39908
I20260620 03:06:15.404917 6566 tablet_copy_service.cc:433] P 85a88abe479747acb7a7878d64c9d24c: ending tablet copy session 5f373e2277904013a4c4acee64771191-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5f373e2277904013a4c4acee64771191
I20260620 03:06:15.415836 7007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:15.418380 7121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:15.420639 7122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:15.442421 7124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:15.444873 7007 server_base.cc:1034] running on GCE node
I20260620 03:06:15.445221 7007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:15.445935 7007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:15.447149 7007 hybrid_clock.cc:648] HybridClock initialized: now 1781924775447092 us; error 35 us; skew 500 ppm
I20260620 03:06:15.447396 7007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:15 dist-test-slave-xv68 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924775, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:15.449601 7007 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 03:06:15.456880 7007 webserver.cc:460] Webserver started at http://127.0.97.61:44505/ using document root <none> and password file <none>
I20260620 03:06:15.457216 7007 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:15.457307 7007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:15.459683 7007 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 03:06:15.461391 7131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:15.461849 7007 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 03:06:15.461999 7007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/wal
uuid: "5f373e2277904013a4c4acee64771191"
format_stamp: "Formatted at 2026-06-20 03:06:15 on dist-test-slave-xv68"
server_key: "83114c564a11e51744731d687d0081b1"
server_key_iv: "4a09c79fcd26e1143cb994e05af232e3"
server_key_version: "encryptionkey@0"
I20260620 03:06:15.462153 7007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:15.477764 7007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:15.481042 7007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:15.510324 7007 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44195
I20260620 03:06:15.510349 7182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44195 every 8 connection(s)
I20260620 03:06:15.511193 7007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-1/data/info.pb
I20260620 03:06:15.518121 7183 sys_catalog.cc:263] Verifying existing consensus state
I20260620 03:06:15.520886 7183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: Bootstrap starting.
I20260620 03:06:15.530834 7183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 03:06:15.531332 7183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: Bootstrap complete.
I20260620 03:06:15.532176 7183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: true } }
I20260620 03:06:15.532405 7183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f373e2277904013a4c4acee64771191, State: Initialized, Role: LEARNER
I20260620 03:06:15.532716 7183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: true } }
I20260620 03:06:15.533576 7186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: true } } }
I20260620 03:06:15.533901 7183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:06:15.533941 7186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: This master's current role is: LEARNER
I20260620 03:06:15.541347 7197 catalog_manager.cc:1259] Loaded cluster ID: eec7f634163546948ee74189cc31cdbb
I20260620 03:06:15.541455 7197 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: loading cluster ID for follower catalog manager: success
I20260620 03:06:15.543943 7197 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: acquiring CA information for follower catalog manager: success
I20260620 03:06:15.545955 7197 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 03:06:15.630816 7002 heartbeater.cc:503] Master 127.0.97.62:45499 was elected leader, sending a full tablet report...
W20260620 03:06:15.706627 7105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:15.706889 7105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:15.707029 7105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:15.714038 7105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:06:15.714133 7105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:15.714229 7105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:15.714324 7105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:06:15.714398 7105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:06:15.724411 7105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45499,127.0.97.61:44195,127.0.97.60:45945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:15.725658 7105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:15.727005 7105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:15.739164 7202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:15.739203 7203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:15.739226 7205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:15.740640 7105 server_base.cc:1034] running on GCE node
I20260620 03:06:15.741333 7105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:15.742772 7105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:15.744010 7105 hybrid_clock.cc:648] HybridClock initialized: now 1781924775743937 us; error 58 us; skew 500 ppm
I20260620 03:06:15.744346 7105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:15 dist-test-slave-xv68 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924775, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:15.747460 7105 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 03:06:15.748904 7105 webserver.cc:460] Webserver started at http://127.0.97.60:37955/ using document root <none> and password file <none>
I20260620 03:06:15.749526 7105 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:15.749621 7105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:15.749931 7105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:15.753104 7105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/data/instance:
uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5"
format_stamp: "Formatted at 2026-06-20 03:06:15 on dist-test-slave-xv68"
server_key: "570b396deb02db91d85a931004c6f02b"
server_key_iv: "74c196d0e5dc14d2b4a7b4b7ac614ae2"
server_key_version: "encryptionkey@0"
I20260620 03:06:15.753883 7105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/wal/instance:
uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5"
format_stamp: "Formatted at 2026-06-20 03:06:15 on dist-test-slave-xv68"
server_key: "570b396deb02db91d85a931004c6f02b"
server_key_iv: "74c196d0e5dc14d2b4a7b4b7ac614ae2"
server_key_version: "encryptionkey@0"
I20260620 03:06:15.758656 7105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 03:06:15.761875 7212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:15.763281 7105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:06:15.763492 7105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/wal
uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5"
format_stamp: "Formatted at 2026-06-20 03:06:15 on dist-test-slave-xv68"
server_key: "570b396deb02db91d85a931004c6f02b"
server_key_iv: "74c196d0e5dc14d2b4a7b4b7ac614ae2"
server_key_version: "encryptionkey@0"
I20260620 03:06:15.763684 7105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:15.778116 7105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:15.781945 7105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:15.782214 7105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:15.801219 7105 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:45945
I20260620 03:06:15.801242 7263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:45945 every 8 connection(s)
I20260620 03:06:15.802526 7105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/data/info.pb
I20260620 03:06:15.804734 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 7105
I20260620 03:06:15.807448 7264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:15.814265 7264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:06:15 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 03:06:15 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924775, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 03:06:15.828222 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.806781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39628 (local address 127.0.97.60:45945)
0620 03:06:15.807572 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:15.807587 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:15.807644 (+ 57us) server_negotiation.cc:407] Connection header received
0620 03:06:15.809306 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:15.809327 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:15.809701 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:15.810091 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:15.811545 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.813188 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.814430 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.815071 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.815266 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.820020 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:15.820027 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:15.820037 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:15.820071 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:15.823664 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.824390 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.824396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.824401 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.824578 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.824998 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.825004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.825008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.825569 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:15.825775 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:15.826705 (+ 930us) server_negotiation.cc:299] Negotiation successful
0620 03:06:15.827096 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":334,"threads_started":1}
I20260620 03:06:15.831779 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.816919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:39916)
0620 03:06:15.817631 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:15.817669 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:15.817848 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:15.818215 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:15.818223 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:15.818783 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:15.819004 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.819012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.820137 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.820143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.821504 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.821519 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.822223 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.822228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.822350 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.822943 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:15.822959 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:15.825316 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:15.829073 (+ 3757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.829081 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.829088 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.829630 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.830020 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.830023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.830025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.830146 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.830826 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:15.830833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:15.831011 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 03:06:15.831223 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 03:06:15.831508 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":372,"threads_started":1}
I20260620 03:06:15.831787 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.816988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.0.97.62:45499)
0620 03:06:15.817147 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:15.817173 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:15.817768 (+ 595us) server_negotiation.cc:407] Connection header received
0620 03:06:15.817988 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:15.817993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:15.818053 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:15.818157 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:15.819230 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.819994 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.821661 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.822057 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.822157 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.825515 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:15.825520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:15.825523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:15.825552 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:15.828953 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.829743 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.829748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.829752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.829908 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.830264 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.830269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.830272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.830659 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:15.830810 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:15.831370 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 03:06:15.831563 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 03:06:15.833109 7264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:06:15 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924775, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 03:06:15.845976 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.833770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44195 (local address 127.0.0.1:49518)
0620 03:06:15.833978 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:15.833994 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:15.834067 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:15.834723 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:15.834727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:15.835086 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:15.835212 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.835219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.836494 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.836498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.837565 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.837576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.838194 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.838198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.838308 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.838827 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:15.838842 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:15.841014 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:15.843894 (+ 2880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.843898 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.843900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.844326 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.844666 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.844669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.844671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.844798 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.845268 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:15.845271 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:15.845445 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 03:06:15.845548 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 03:06:15.845758 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 03:06:15.846052 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.833895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.0.97.61:44195)
0620 03:06:15.834394 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:15.834398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:15.834413 (+ 15us) server_negotiation.cc:407] Connection header received
0620 03:06:15.834490 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:15.834493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:15.834551 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:15.834641 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:15.835377 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.836349 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.837682 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.838070 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.838161 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.841199 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:15.841204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:15.841207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:15.841237 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:15.843778 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.844426 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.844430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.844461 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.844576 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.844910 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.844914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.844916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.845171 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:15.845349 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:15.845639 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 03:06:15.845784 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":289,"threads_started":1}
I20260620 03:06:15.847594 7264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:06:15 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924775, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 03:06:15.859118 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.848235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45945 (local address 127.0.0.1:39640)
0620 03:06:15.848383 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:15.848397 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:15.848477 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:15.848707 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:15.848709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:15.849005 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:15.849131 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.849137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.850165 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.850171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.851033 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.851046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.851768 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.851773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.851922 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.852402 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:15.852429 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:15.854308 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:15.857147 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.857151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.857153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.857500 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.857807 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.857811 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.857813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.857950 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.858388 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:15.858391 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:15.858536 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 03:06:15.858652 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 03:06:15.858824 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 03:06:15.859274 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.848379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39640 (local address 127.0.97.60:45945)
0620 03:06:15.848519 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:15.848522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:15.848536 (+ 14us) server_negotiation.cc:407] Connection header received
0620 03:06:15.848576 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:15.848579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:15.848628 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:15.848697 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:15.849270 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.849994 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.851160 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.851597 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.851705 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.854451 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:15.854456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:15.854459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:15.854492 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:15.857011 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.857598 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.857601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.857603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.857716 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.858047 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.858050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.858052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.858270 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:15.858368 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:15.858752 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 03:06:15.858931 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":52}
I20260620 03:06:15.864039 7264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: Bootstrap starting.
I20260620 03:06:15.867100 7264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:15.868568 7264 log.cc:824] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:15.870975 7264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: No bootstrap required, opened a new log
I20260620 03:06:15.877734 7264 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } }
I20260620 03:06:15.878064 7264 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:15.878188 7264 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ed128efced40ed8d9d0b9fd4ada0f5, State: Initialized, Role: FOLLOWER
I20260620 03:06:15.878705 7264 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } }
I20260620 03:06:15.879544 7271 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } } }
I20260620 03:06:15.879827 7271 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:06:15.880445 7264 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:06:15.884119 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: true } } }
I20260620 03:06:15.884545 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:15.896152 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.884567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49522 (local address 127.0.97.61:44195)
0620 03:06:15.884787 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:15.884796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:15.888418 (+ 3622us) server_negotiation.cc:407] Connection header received
0620 03:06:15.888718 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:15.888724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:15.888843 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:15.888940 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:15.889514 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.890479 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.893930 (+ 3451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.894855 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.895025 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.895209 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:15.895745 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 03:06:15.895886 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 03:06:15.896765 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.884226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44195 (local address 127.0.0.1:49522)
0620 03:06:15.888287 (+ 4061us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:15.888321 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:15.888505 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:15.889163 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:15.889166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:15.889188 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:15.889360 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.889367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.890637 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.890642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.893654 (+ 3012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.893677 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.895342 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.895348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.895541 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.895598 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 03:06:15.896322 (+ 724us) client_negotiation.cc:240] Negotiation successful
0620 03:06:15.896468 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3912,"thread_start_us":3793,"threads_started":1}
I20260620 03:06:15.898022 7186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85a88abe479747acb7a7878d64c9d24c. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: true } } }
I20260620 03:06:15.898252 7186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: This master's current role is: LEARNER
W20260620 03:06:15.898219 7287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 03:06:15.898340 7287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 03:06:15.899071 7105 master_runner.cc:428] Detected that this master d7ed128efced40ed8d9d0b9fd4ada0f5 is joining an existing cluster
I20260620 03:06:15.899186 7105 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:45945
I20260620 03:06:15.899191 7277 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c: attempting to promote NON_VOTER 5f373e2277904013a4c4acee64771191 to VOTER
I20260620 03:06:15.899956 7277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } }
I20260620 03:06:15.901465 7158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 1 LEARNER]: Refusing update from remote peer 85a88abe479747acb7a7878d64c9d24c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 03:06:15.902038 7278 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 03:06:15.905205 7277 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5f373e2277904013a4c4acee64771191 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } }
I20260620 03:06:15.905851 7158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5f373e2277904013a4c4acee64771191 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } }
I20260620 03:06:15.909183 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } }
I20260620 03:06:15.909144 7284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } }
I20260620 03:06:15.909520 7284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:15.909521 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:15.910264 7186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } }
I20260620 03:06:15.910481 7186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:06:15.935663 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 6607
I20260620 03:06:15.939400 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.915897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.0.97.61:44195)
0620 03:06:15.916062 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:15.916069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:15.921187 (+ 5118us) server_negotiation.cc:407] Connection header received
0620 03:06:15.921456 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:15.921463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:15.921546 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:15.921646 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:15.923490 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.924611 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.927560 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.928199 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.928300 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.931559 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:15.931565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:15.931569 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:15.931616 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:15.935740 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.936560 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.936568 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.936571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.936736 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.937210 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.937218 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.937223 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.937586 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:15.937742 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:15.938549 (+ 807us) server_negotiation.cc:299] Negotiation successful
0620 03:06:15.938872 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 03:06:15.940685 7294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.916762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44195 (local address 127.0.0.1:49534)
0620 03:06:15.921069 (+ 4307us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:15.921093 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:15.921299 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:15.921821 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:15.921825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:15.922795 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:15.923331 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.923339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.924801 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.924806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.927402 (+ 2596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.927416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.928665 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.928672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.928961 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.929827 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:15.929861 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:15.931369 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:15.935895 (+ 4526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.935900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.935903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.936427 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.936884 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.936889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.936897 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.937086 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.937841 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:15.937849 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:15.938137 (+ 288us) client_negotiation.cc:769] Sending connection context
0620 03:06:15.939655 (+ 1518us) client_negotiation.cc:240] Negotiation successful
0620 03:06:15.940000 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4135,"mutex_wait_us":314,"spinlock_wait_cycles":26496,"thread_start_us":3420,"threads_started":1}
I20260620 03:06:15.941733 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.916683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39924 (local address 127.0.97.62:45499)
0620 03:06:15.916842 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:15.916848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:15.921415 (+ 4567us) server_negotiation.cc:407] Connection header received
0620 03:06:15.921869 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:15.921876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:15.921955 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:15.922037 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:15.924137 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.925227 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.927573 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.928255 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.928355 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.931626 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:15.931632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:15.931637 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:15.931675 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:15.938590 (+ 6915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.939365 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.939371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.939375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.939526 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.939957 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.939962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.939965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.940343 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:15.940479 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:15.941181 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 03:06:15.941385 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 03:06:15.942947 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.917135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:39924)
0620 03:06:15.921325 (+ 4190us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:15.921341 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:15.921708 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:15.922265 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:15.922270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:15.923008 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:15.923948 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.923961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.925400 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.925406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.927408 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.927421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.928682 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.928689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.928944 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.929725 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:15.929757 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:15.931382 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:15.938741 (+ 7359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.938747 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.938751 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.939208 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.939676 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.939681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.939684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.939834 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.940564 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:15.940570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:15.940790 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 03:06:15.942101 (+ 1311us) client_negotiation.cc:240] Negotiation successful
0620 03:06:15.942408 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4077,"mutex_wait_us":122,"thread_start_us":4297,"threads_started":1}
I20260620 03:06:15.948532 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.915562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.0.97.60:45945)
0620 03:06:15.915848 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:15.915855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:15.920618 (+ 4763us) server_negotiation.cc:407] Connection header received
0620 03:06:15.921639 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:15.921643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:15.921705 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:15.921764 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:15.923470 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.924792 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.927547 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.928212 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.928300 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.931554 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:15.931560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:15.931564 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:15.931596 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:15.937002 (+ 5406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.945812 (+ 8810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.945818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.945821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.945957 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.946821 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.946827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.946830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.947175 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:15.947359 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:15.947980 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 03:06:15.948185 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":60}
I20260620 03:06:15.949682 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.915529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45945 (local address 127.0.0.1:39656)
0620 03:06:15.920513 (+ 4984us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:15.920539 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:15.921509 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:15.921951 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:15.921956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:15.922773 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:15.923319 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.923332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.924980 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.924986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.927415 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.927423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.928595 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.928606 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.928840 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.929937 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:15.929959 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:15.931385 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:15.943381 (+ 11996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.943389 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.943396 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.944138 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.946489 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.946497 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.946501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.946693 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.947573 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:15.947578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:15.947747 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 03:06:15.948887 (+ 1140us) client_negotiation.cc:240] Negotiation successful
0620 03:06:15.949197 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4837,"mutex_wait_us":218,"thread_start_us":5195,"threads_started":1}
Jun 20 03:06:15 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
Jun 20 03:06:15 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924772, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 03:06:15.966169 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.948526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45945 (local address 127.0.0.1:39668)
0620 03:06:15.948708 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:15.948732 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:15.948832 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:15.949808 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:15.949813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:15.950326 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:15.950531 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.950543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.952259 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.952264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.953535 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:15.953546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.954694 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.954702 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:15.954908 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.956021 (+ 1113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:15.956047 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:15.959011 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:15.963446 (+ 4435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.963451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.963455 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.964005 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.964389 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:15.964396 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:15.964401 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:15.964557 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:15.965163 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:15.965168 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:15.965364 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 03:06:15.965508 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 03:06:15.965778 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 03:06:15.966396 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.937534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.0.97.1:38365)
0620 03:06:15.944738 (+ 7204us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:15.944752 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:15.944775 (+ 23us) server_negotiation.cc:407] Connection header received
0620 03:06:15.944828 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:15.944834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:15.945040 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:15.945214 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:15.947407 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.949411 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.951520 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.952790 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.952927 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.958707 (+ 5780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:15.958719 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:15.958733 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:15.958770 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:15.962789 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.963726 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.963735 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.963739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.963882 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.964360 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.964366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.964371 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.964773 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:15.965041 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:15.965570 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 03:06:15.965824 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7035,"thread_start_us":6920,"threads_started":1}
I20260620 03:06:15.967144 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:15.949135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.0.97.60:45945)
0620 03:06:15.949400 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:15.949407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:15.949433 (+ 26us) server_negotiation.cc:407] Connection header received
0620 03:06:15.949527 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:15.949532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:15.949635 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:15.950718 (+ 1083us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:15.950906 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.952136 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.953780 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:15.954502 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:15.954641 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:15.959219 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:15.959225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:15.959232 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:15.959270 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:15.963272 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.964138 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.964143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.964146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.964275 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:15.964684 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:15.964690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:15.964693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:15.965026 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:15.965163 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:15.966574 (+ 1411us) server_negotiation.cc:299] Negotiation successful
0620 03:06:15.966789 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260620 03:06:15.967796 6730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:15.969355 6546 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:45945: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 45945
}
attrs {
promote: true
}
}
dest_uuid: "85a88abe479747acb7a7878d64c9d24c"
cas_config_opid_index: 6
I20260620 03:06:15.970131 6546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: true } }
I20260620 03:06:15.972512 7158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 1 FOLLOWER]: Refusing update from remote peer 85a88abe479747acb7a7878d64c9d24c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 03:06:15.972829 7239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:15.975642 7284 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 03:06:15.978281 7277 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d7ed128efced40ed8d9d0b9fd4ada0f5 (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: true } } }
I20260620 03:06:15.978840 7158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d7ed128efced40ed8d9d0b9fd4ada0f5 (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: true } } }
I20260620 03:06:15.983367 7186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: true } } }
I20260620 03:06:15.983755 7186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:06:15.984078 7277 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:45945
I20260620 03:06:15.984747 7105 master.cc:537] Master@127.0.97.60:45945 shutting down...
I20260620 03:06:15.984581 7278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: true } } }
I20260620 03:06:15.984947 7278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:15.986218 7278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: true } } }
I20260620 03:06:15.986536 7278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:15.986886 7239 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 1 FOLLOWER]: Refusing update from remote peer 85a88abe479747acb7a7878d64c9d24c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 03:06:15.987874 7277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 03:06:15.991284 6530 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 03:06:15.992974 6530 consensus_peers.cc:487] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c -> Peer d7ed128efced40ed8d9d0b9fd4ada0f5 (127.0.97.60:45945): Couldn't send request to peer d7ed128efced40ed8d9d0b9fd4ada0f5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 03:06:16.016575 7105 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:06:16.017004 7105 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:16.017208 7105 tablet_replica.cc:330] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: stopping tablet replica
I20260620 03:06:16.024214 7105 master.cc:552] Master@127.0.97.60:45945 shutdown complete.
I20260620 03:06:16.024305 7105 master_runner.cc:305] Clearing existing system tablet
I20260620 03:06:16.024679 7105 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:16.024780 7105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:16.027608 7105 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260620 03:06:16.029259 7301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:16.029625 7105 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:16.029939 7105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 03:06:16.030066 7105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/wal
uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5"
format_stamp: "Formatted at 2026-06-20 03:06:15 on dist-test-slave-xv68"
server_key: "570b396deb02db91d85a931004c6f02b"
server_key_iv: "74c196d0e5dc14d2b4a7b4b7ac614ae2"
server_key_version: "encryptionkey@0"
I20260620 03:06:16.035247 7105 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:16.042503 7105 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 03:06:16.042699 7105 log.cc:1196] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 03:06:16.043212 7105 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: Deleting consensus metadata
I20260620 03:06:16.043525 7105 master_runner.cc:315] Copying system tablet from 127.0.97.62:45499
I20260620 03:06:16.046001 7105 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:45499
I20260620 03:06:16.066920 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:16.046894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:39932)
0620 03:06:16.052288 (+ 5394us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:16.052311 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:16.052483 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:16.052897 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:16.052902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:16.053441 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:16.053700 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:16.053712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.055199 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.055204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:16.056728 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:16.056743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.057814 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.057821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:16.058014 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:16.058784 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:16.058810 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:16.059886 (+ 1076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:16.064084 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:16.064089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:16.064094 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:16.064568 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:16.064997 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:16.065002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:16.065006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:16.065167 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:16.065811 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:16.065816 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:16.066014 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 03:06:16.066152 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 03:06:16.066452 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5255,"thread_start_us":4629,"threads_started":1}
I20260620 03:06:16.066968 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:16.046926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39932 (local address 127.0.97.62:45499)
0620 03:06:16.047145 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:16.047151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:16.052442 (+ 5291us) server_negotiation.cc:407] Connection header received
0620 03:06:16.052646 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:16.052653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:16.052739 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:16.052851 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:16.053850 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.055037 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.056887 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.057645 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.057799 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:16.060101 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:16.060108 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:16.060114 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:16.060158 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:16.063953 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:16.064715 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:16.064721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:16.064726 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:16.064880 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:16.065292 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:16.065298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:16.065302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:16.065672 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:16.065826 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:16.066311 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 03:06:16.066558 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 03:06:16.068063 6566 tablet_copy_service.cc:140] P 85a88abe479747acb7a7878d64c9d24c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d7ed128efced40ed8d9d0b9fd4ada0f5 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:39932)
I20260620 03:06:16.068249 6566 tablet_copy_service.cc:161] P 85a88abe479747acb7a7878d64c9d24c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d7ed128efced40ed8d9d0b9fd4ada0f5 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:39932: session id = d7ed128efced40ed8d9d0b9fd4ada0f5-00000000000000000000000000000000
I20260620 03:06:16.072713 6566 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 03:06:16.074584 7105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 03:06:16.081125 7105 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: tablet copy: Starting download of 0 data blocks...
I20260620 03:06:16.081390 7105 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: tablet copy: Starting download of 1 WAL segments...
I20260620 03:06:16.083731 7105 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 03:06:16.088837 6566 tablet_copy_service.cc:342] P 85a88abe479747acb7a7878d64c9d24c: Request end of tablet copy session d7ed128efced40ed8d9d0b9fd4ada0f5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:39932
I20260620 03:06:16.088984 6566 tablet_copy_service.cc:433] P 85a88abe479747acb7a7878d64c9d24c: ending tablet copy session d7ed128efced40ed8d9d0b9fd4ada0f5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d7ed128efced40ed8d9d0b9fd4ada0f5
I20260620 03:06:16.099408 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 6607
I20260620 03:06:16.102145 7105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:16.104811 7308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:16.116319 7309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:16.123670 7311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:16.123997 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:38365
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45499
--webserver_port=39691
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:45499,127.0.97.61:44195
--tserver_master_addrs=127.0.97.62:45499,127.0.97.61:44195,127.0.97.60:45945 with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:06:16.124730 7105 server_base.cc:1034] running on GCE node
I20260620 03:06:16.125030 7105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:16.125696 7105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:16.133069 7105 hybrid_clock.cc:648] HybridClock initialized: now 1781924776133017 us; error 40 us; skew 500 ppm
I20260620 03:06:16.133313 7105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:16 dist-test-slave-xv68 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:16.135411 7105 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 03:06:16.140380 7105 webserver.cc:460] Webserver started at http://127.0.97.60:46781/ using document root <none> and password file <none>
I20260620 03:06:16.140733 7105 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:16.140846 7105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:16.147956 7105 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.004s
I20260620 03:06:16.149624 7319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:16.150111 7105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 03:06:16.150241 7105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/wal
uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5"
format_stamp: "Formatted at 2026-06-20 03:06:15 on dist-test-slave-xv68"
server_key: "570b396deb02db91d85a931004c6f02b"
server_key_iv: "74c196d0e5dc14d2b4a7b4b7ac614ae2"
server_key_version: "encryptionkey@0"
I20260620 03:06:16.150444 7105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:16.164888 7105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:16.168222 7105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:16.187906 7105 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:45945
I20260620 03:06:16.187963 7370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:45945 every 8 connection(s)
I20260620 03:06:16.188839 7105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/master-2/data/info.pb
I20260620 03:06:16.196101 7371 sys_catalog.cc:263] Verifying existing consensus state
I20260620 03:06:16.200742 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: Bootstrap starting.
I20260620 03:06:16.216696 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 03:06:16.217173 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: Bootstrap complete.
I20260620 03:06:16.218077 7371 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: true } }
I20260620 03:06:16.218351 7371 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7ed128efced40ed8d9d0b9fd4ada0f5, State: Initialized, Role: LEARNER
I20260620 03:06:16.218573 7371 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: true } }
I20260620 03:06:16.219751 7374 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: true } } }
I20260620 03:06:16.220077 7371 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:06:16.220222 7374 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [sys.catalog]: This master's current role is: LEARNER
I20260620 03:06:16.237920 7385 catalog_manager.cc:1259] Loaded cluster ID: eec7f634163546948ee74189cc31cdbb
I20260620 03:06:16.238036 7385 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: loading cluster ID for follower catalog manager: success
I20260620 03:06:16.241138 7385 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: acquiring CA information for follower catalog manager: success
I20260620 03:06:16.243230 7385 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 03:06:16.511622 7314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:16.511960 7314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:16.512095 7314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:16.523610 7314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:16.523814 7314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:16.524026 7314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:06:16.534718 7390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:16.519492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39682 (local address 127.0.97.60:45945)
0620 03:06:16.528136 (+ 8644us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:16.528140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:16.528159 (+ 19us) server_negotiation.cc:407] Connection header received
0620 03:06:16.528219 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:16.528223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:16.528291 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:16.528396 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:16.528826 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.529831 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.532572 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.533724 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.533833 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:16.533983 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:16.534288 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 03:06:16.534382 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8540,"thread_start_us":8380,"threads_started":1}
I20260620 03:06:16.534812 7389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:16.519347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45945 (local address 127.0.0.1:39682)
0620 03:06:16.527791 (+ 8444us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:16.527813 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:16.527965 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:16.528448 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:16.528452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:16.528482 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:16.528683 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:16.528693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.530038 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.530044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:16.532392 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:16.532408 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.533911 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.533917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:16.534131 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:16.534167 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 03:06:16.534258 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 03:06:16.534427 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8289,"thread_start_us":8197,"threads_started":1}
I20260620 03:06:16.536145 7346 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260620 03:06:16.536684 7374 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85a88abe479747acb7a7878d64c9d24c. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: true } } }
I20260620 03:06:16.537009 7374 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [sys.catalog]: This master's current role is: LEARNER
I20260620 03:06:16.537959 7314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:38365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=39691
--tserver_master_addrs=127.0.97.62:45499,127.0.97.61:44195,127.0.97.60:45945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:16.539167 7314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:16.540503 7314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:16.554536 7395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:16.554564 7393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:16.554627 7392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:16.560681 7314 server_base.cc:1034] running on GCE node
I20260620 03:06:16.561254 7314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:16.562211 7314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:16.563388 7314 hybrid_clock.cc:648] HybridClock initialized: now 1781924776563327 us; error 69 us; skew 500 ppm
I20260620 03:06:16.563699 7314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:16 dist-test-slave-xv68 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:16.566707 7314 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 03:06:16.568233 7314 webserver.cc:460] Webserver started at http://127.0.97.1:39691/ using document root <none> and password file <none>
I20260620 03:06:16.568920 7314 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:16.569072 7314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:16.574142 7314 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 03:06:16.577342 7402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:16.578555 7314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 03:06:16.578742 7314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "63870f6d79a144b7bd7696c4c4c47b4d"
format_stamp: "Formatted at 2026-06-20 03:06:13 on dist-test-slave-xv68"
server_key: "4ee44307c91558d9c3c4572f6aec2e1d"
server_key_iv: "c8e22aec13bd136b48060713f5a52b51"
server_key_version: "encryptionkey@0"
I20260620 03:06:16.579734 7314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:16.597674 7314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:16.601817 7314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:16.602077 7314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:16.603032 7314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:16.604724 7314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:16.604842 7314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:16.604980 7314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:16.605067 7314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:16.639880 7314 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38365
I20260620 03:06:16.639925 7514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38365 every 8 connection(s)
I20260620 03:06:16.641343 7314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:06:16.660055 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 7314
I20260620 03:06:16.660432 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 6741
Jun 20 03:06:16 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Jun 20 03:06:16 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 03:06:16 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 03:06:16 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 03:06:16.698081 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:16.661742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56566 (local address 127.0.97.2:35015)
0620 03:06:16.674760 (+ 13018us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:16.674772 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:16.674809 (+ 37us) server_negotiation.cc:407] Connection header received
0620 03:06:16.674897 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:16.674908 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:16.675270 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:16.676929 (+ 1659us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:16.680896 (+ 3967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.681844 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.683711 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.684521 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.684625 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:16.690295 (+ 5670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:16.690306 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:16.690324 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:16.690354 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:16.694029 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:16.694883 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:16.694894 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:16.694897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:16.695071 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:16.695562 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:16.695569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:16.695574 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:16.695998 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:16.696284 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:16.697386 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0620 03:06:16.697638 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12808,"thread_start_us":12648,"threads_started":1}
I20260620 03:06:16.699574 6864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:16.701360 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:16.645471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57999 (local address 127.0.97.61:44195)
0620 03:06:16.677784 (+ 32313us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:16.677788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:16.677809 (+ 21us) server_negotiation.cc:407] Connection header received
0620 03:06:16.677852 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:16.677856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:16.677910 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:16.678032 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:16.680405 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.681362 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.683888 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.684478 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.684599 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:16.693579 (+ 8980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:16.693586 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:16.693590 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:16.693629 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:16.697515 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:16.699028 (+ 1513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:16.699033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:16.699036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:16.699175 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:16.699648 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:16.699653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:16.699657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:16.700005 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:16.700135 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:16.700767 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 03:06:16.700975 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32214,"thread_start_us":32120,"threads_started":1}
I20260620 03:06:16.701889 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:16.644884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40293 (local address 127.0.97.62:45499)
0620 03:06:16.676692 (+ 31808us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:16.676701 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:16.680445 (+ 3744us) server_negotiation.cc:407] Connection header received
0620 03:06:16.680508 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:16.680513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:16.680584 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:16.680687 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:16.682917 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.684648 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.686429 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.687162 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.687338 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:16.693283 (+ 5945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:16.693291 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:16.693296 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:16.693335 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:16.696859 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:16.698584 (+ 1725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:16.698588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:16.698592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:16.698723 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:16.699352 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:16.699357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:16.699360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:16.699677 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:16.699813 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:16.700797 (+ 984us) server_negotiation.cc:299] Negotiation successful
0620 03:06:16.700984 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31646,"thread_start_us":38016,"threads_started":1}
I20260620 03:06:16.703780 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:16.644919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44195 (local address 127.0.97.1:57999)
0620 03:06:16.676177 (+ 31258us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:16.676196 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:16.676313 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:16.678305 (+ 1992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:16.678317 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:16.679337 (+ 1020us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:16.680170 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:16.680204 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.681535 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.681539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:16.683742 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:16.683755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.684971 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.684979 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:16.685309 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:16.689101 (+ 3792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:16.689136 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:16.693379 (+ 4243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:16.697614 (+ 4235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:16.697620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:16.697625 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:16.698899 (+ 1274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:16.699331 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:16.699337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:16.699343 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:16.699509 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:16.700189 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:16.700193 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:16.700305 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 03:06:16.702046 (+ 1741us) client_negotiation.cc:240] Negotiation successful
0620 03:06:16.702626 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31142,"mutex_wait_us":70,"spinlock_wait_cycles":18176,"thread_start_us":30943,"threads_started":1}
I20260620 03:06:16.703830 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:16.644607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:40293)
0620 03:06:16.680163 (+ 35556us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:16.680185 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:16.680330 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:16.682000 (+ 1670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:16.682003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:16.682501 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:16.682782 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:16.682790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.684753 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.684758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:16.686260 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:16.686274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.687389 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.687396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:16.687596 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:16.689259 (+ 1663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:16.689276 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:16.693020 (+ 3744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:16.697640 (+ 4620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:16.697644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:16.697647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:16.698435 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:16.699012 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:16.699016 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:16.699023 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:16.699165 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:16.699896 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:16.699904 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:16.700279 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 03:06:16.702029 (+ 1750us) client_negotiation.cc:240] Negotiation successful
0620 03:06:16.702839 (+ 810us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35354,"mutex_wait_us":689,"spinlock_wait_cycles":896,"thread_start_us":30244,"threads_started":1}
I20260620 03:06:16.705639 7517 heartbeater.cc:347] Connected to a master server at 127.0.97.61:44195
I20260620 03:06:16.705634 7516 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45499
I20260620 03:06:16.705964 7517 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:16.705976 7516 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:16.707011 7516 heartbeater.cc:511] Master 127.0.97.62:45499 requested a full tablet report, sending...
I20260620 03:06:16.707042 7517 heartbeater.cc:511] Master 127.0.97.61:44195 requested a full tablet report, sending...
I20260620 03:06:16.708084 7390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:16.643815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45197 (local address 127.0.97.60:45945)
0620 03:06:16.644021 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:16.644027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:16.669143 (+ 25116us) server_negotiation.cc:407] Connection header received
0620 03:06:16.683114 (+ 13971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:16.683120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:16.683200 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:16.683271 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:16.685228 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.686286 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.687616 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.688295 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.688455 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:16.693836 (+ 5381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:16.693844 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:16.693848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:16.693884 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:16.697409 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:16.698558 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:16.698563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:16.698567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:16.698689 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:16.706312 (+ 7623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:16.706318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:16.706322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:16.706683 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:16.706832 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:16.707524 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 03:06:16.707746 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 03:06:16.708907 6546 ts_manager.cc:194] Re-registered known tserver with Master: 63870f6d79a144b7bd7696c4c4c47b4d (127.0.97.1:38365)
I20260620 03:06:16.709124 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:16.644606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45945 (local address 127.0.97.1:45197)
0620 03:06:16.668897 (+ 24291us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:16.668967 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:16.672539 (+ 3572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:16.683503 (+ 10964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:16.683510 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:16.684399 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:16.684609 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:16.684619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.686446 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.686449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:16.687419 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:16.687436 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:16.688483 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:16.688488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:16.688608 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:16.689504 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:16.689524 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:16.693631 (+ 4107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:16.697567 (+ 3936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:16.697577 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:16.697592 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:16.698432 (+ 840us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:16.698798 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:16.698804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:16.698808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:16.699107 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:16.707022 (+ 7915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:16.707032 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:16.707199 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 03:06:16.708436 (+ 1237us) client_negotiation.cc:240] Negotiation successful
0620 03:06:16.708740 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23825,"spinlock_wait_cycles":22272,"thread_start_us":35269,"threads_started":1}
I20260620 03:06:16.709362 7148 ts_manager.cc:194] Registered new tserver with Master: 63870f6d79a144b7bd7696c4c4c47b4d (127.0.97.1:38365)
I20260620 03:06:16.709915 6546 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40293
I20260620 03:06:16.711402 7515 heartbeater.cc:347] Connected to a master server at 127.0.97.60:45945
I20260620 03:06:16.711535 7515 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:16.712181 7515 heartbeater.cc:511] Master 127.0.97.60:45945 requested a full tablet report, sending...
I20260620 03:06:16.713582 7336 ts_manager.cc:194] Registered new tserver with Master: 63870f6d79a144b7bd7696c4c4c47b4d (127.0.97.1:38365)
I20260620 03:06:16.787712 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 6741
I20260620 03:06:16.803736 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35015
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45499
--webserver_port=41637
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:45499,127.0.97.61:44195
--tserver_master_addrs=127.0.97.62:45499,127.0.97.61:44195,127.0.97.60:45945 with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:06:16.986178 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: true } } }
I20260620 03:06:16.986598 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:16.988286 7277 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c: attempting to promote NON_VOTER d7ed128efced40ed8d9d0b9fd4ada0f5 to VOTER
I20260620 03:06:16.988801 7277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } }
I20260620 03:06:17.000451 7346 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 1 LEARNER]: Refusing update from remote peer 85a88abe479747acb7a7878d64c9d24c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 03:06:17.000459 7158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 1 FOLLOWER]: Refusing update from remote peer 85a88abe479747acb7a7878d64c9d24c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 03:06:17.001149 7529 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 03:06:17.001621 7528 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 03:06:17.016482 7529 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d7ed128efced40ed8d9d0b9fd4ada0f5 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } } }
I20260620 03:06:17.017354 7345 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d7ed128efced40ed8d9d0b9fd4ada0f5 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } } }
I20260620 03:06:17.020776 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } } }
I20260620 03:06:17.021013 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:17.021379 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } } }
I20260620 03:06:17.021692 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:17.022331 7374 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } } }
I20260620 03:06:17.022567 7374 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:06:17.023015 7528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } } }
I20260620 03:06:17.023324 7528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:17.023676 7158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d7ed128efced40ed8d9d0b9fd4ada0f5 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } } }
I20260620 03:06:17.026505 7534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85a88abe479747acb7a7878d64c9d24c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } } }
I20260620 03:06:17.026949 7534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 03:06:17.142267 7527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:17.142547 7527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:17.142653 7527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:17.150400 7527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:17.150506 7527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:17.150678 7527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:06:17.160591 7527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41637
--tserver_master_addrs=127.0.97.62:45499,127.0.97.61:44195,127.0.97.60:45945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:17.161808 7527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:17.163127 7527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:17.176301 7539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:17.176368 7542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:17.176403 7540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:17.177763 7527 server_base.cc:1034] running on GCE node
I20260620 03:06:17.178407 7527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:17.179416 7527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:17.180655 7527 hybrid_clock.cc:648] HybridClock initialized: now 1781924777180602 us; error 49 us; skew 500 ppm
I20260620 03:06:17.180996 7527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:17 dist-test-slave-xv68 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:17.184213 7527 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 03:06:17.185627 7527 webserver.cc:460] Webserver started at http://127.0.97.2:41637/ using document root <none> and password file <none>
I20260620 03:06:17.186316 7527 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:17.186476 7527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:17.191926 7527 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:06:17.194784 7549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:17.196278 7527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:06:17.196452 7527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "b56f067b01d34ef1b8c339beaf862a84"
format_stamp: "Formatted at 2026-06-20 03:06:14 on dist-test-slave-xv68"
server_key: "dcdaaa89b00cef25616b0084fc08c69e"
server_key_iv: "cf2af04c8996eae1794da1d2912f2803"
server_key_version: "encryptionkey@0"
I20260620 03:06:17.197427 7527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:17.209753 7527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:17.213979 7527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:17.214250 7527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:17.215261 7527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:17.216603 7527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:17.216686 7527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:17.216763 7527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:17.216816 7527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:17.250344 7527 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35015
I20260620 03:06:17.250389 7661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35015 every 8 connection(s)
I20260620 03:06:17.251979 7527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:06:17.256366 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 7527
I20260620 03:06:17.256711 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 6874
Jun 20 03:06:17 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 03:06:17 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 03:06:17 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Jun 20 03:06:17 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 03:06:17.318192 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.255048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40595 (local address 127.0.97.62:45499)
0620 03:06:17.294896 (+ 39848us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:17.294906 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:17.294934 (+ 28us) server_negotiation.cc:407] Connection header received
0620 03:06:17.296380 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:17.296388 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:17.296497 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:17.296628 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:17.300198 (+ 3570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.301337 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.304143 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.304863 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.304968 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.310315 (+ 5347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:17.310323 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:17.310328 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:17.310368 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:17.315106 (+ 4738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.315884 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.315889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.315892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.316063 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.316526 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.316531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.316534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.316869 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:17.316995 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:17.317595 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 03:06:17.317825 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39687,"thread_start_us":39892,"threads_started":1}
I20260620 03:06:17.319466 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.255744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:40595)
0620 03:06:17.288308 (+ 32564us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:17.288395 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:17.296198 (+ 7803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:17.297143 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:17.297161 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:17.299747 (+ 2586us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:17.300010 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:17.300023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.301572 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.301578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:17.303967 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:17.303983 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.305285 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.305293 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:17.305725 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.306736 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:17.306784 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:17.310058 (+ 3274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:17.315243 (+ 5185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:17.315249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:17.315255 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:17.315748 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:17.316208 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:17.316212 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:17.316216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:17.316404 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:17.317083 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:17.317087 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:17.317260 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 03:06:17.318700 (+ 1440us) client_negotiation.cc:240] Negotiation successful
0620 03:06:17.318972 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32071,"mutex_wait_us":366,"spinlock_wait_cycles":7168,"thread_start_us":41856,"threads_started":1}
I20260620 03:06:17.319932 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.258872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43930 (local address 127.0.97.3:35005)
0620 03:06:17.292921 (+ 34049us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:17.292936 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:17.295235 (+ 2299us) server_negotiation.cc:407] Connection header received
0620 03:06:17.295377 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:17.295391 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:17.295752 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:17.295993 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:17.298830 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.300222 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.302703 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.303609 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.303715 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.311471 (+ 7756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:17.311487 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:17.311505 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:17.311547 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:17.316030 (+ 4483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.316891 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.316899 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.316903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.317281 (+ 378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.317779 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.317785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.317788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.318385 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:17.318630 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:17.319231 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 03:06:17.319503 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33775,"thread_start_us":34084,"threads_started":1}
I20260620 03:06:17.320122 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.255210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45093 (local address 127.0.97.60:45945)
0620 03:06:17.296630 (+ 41420us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:17.296636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:17.296655 (+ 19us) server_negotiation.cc:407] Connection header received
0620 03:06:17.296717 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:17.296724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:17.296813 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:17.296937 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:17.299619 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.300879 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.304164 (+ 3285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.304853 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.304992 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.311866 (+ 6874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:17.311873 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:17.311878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:17.311920 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:17.316025 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.316905 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.316909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.316913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.317069 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.317523 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.317531 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.317536 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.317880 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:17.318018 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:17.319522 (+ 1504us) server_negotiation.cc:299] Negotiation successful
0620 03:06:17.319744 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41272,"thread_start_us":41168,"threads_started":1}
I20260620 03:06:17.320704 7669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.255744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45945 (local address 127.0.97.2:45093)
0620 03:06:17.294065 (+ 38321us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:17.294097 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:17.295737 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:17.297172 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:17.297177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:17.298383 (+ 1206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:17.299376 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:17.299410 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.301127 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.301133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:17.303983 (+ 2850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:17.303993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.305285 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.305293 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:17.305729 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.306736 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:17.306783 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:17.311653 (+ 4870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:17.316185 (+ 4532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:17.316190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:17.316193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:17.316772 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:17.317213 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:17.317222 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:17.317225 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:17.317400 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:17.318077 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:17.318082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:17.318272 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 03:06:17.318427 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 03:06:17.318706 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38124,"mutex_wait_us":306,"spinlock_wait_cycles":45952,"thread_start_us":35439,"threads_started":1}
I20260620 03:06:17.321182 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.257456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34851 (local address 127.0.97.61:44195)
0620 03:06:17.299167 (+ 41711us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:17.299175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:17.299845 (+ 670us) server_negotiation.cc:407] Connection header received
0620 03:06:17.299908 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:17.299913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:17.299992 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:17.300077 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:17.301358 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.302969 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.304982 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.305636 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.305765 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.310670 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:17.310677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:17.310682 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:17.310720 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:17.314230 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.315197 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.315203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.315210 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.315384 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.315883 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.315888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.315891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.316288 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:17.316435 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:17.319558 (+ 3123us) server_negotiation.cc:299] Negotiation successful
0620 03:06:17.319795 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41564,"thread_start_us":41701,"threads_started":1}
I20260620 03:06:17.320845 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.255745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44195 (local address 127.0.97.2:34851)
0620 03:06:17.299614 (+ 43869us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:17.299633 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:17.299744 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:17.300267 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:17.300274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:17.300858 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:17.301140 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:17.301151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.303119 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.303124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:17.304818 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:17.304833 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.306055 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.306063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:17.306318 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.306910 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:17.306934 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:17.310395 (+ 3461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:17.314396 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:17.314417 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:17.314435 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:17.315045 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:17.315527 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:17.315532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:17.315536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:17.315765 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:17.316515 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:17.316523 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:17.316965 (+ 442us) client_negotiation.cc:769] Sending connection context
0620 03:06:17.317299 (+ 334us) client_negotiation.cc:240] Negotiation successful
0620 03:06:17.317926 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43695,"mutex_wait_us":237,"thread_start_us":38317,"threads_started":1}
I20260620 03:06:17.321322 7663 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45499
I20260620 03:06:17.321658 7663 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:17.322219 6997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:17.322629 7662 heartbeater.cc:347] Connected to a master server at 127.0.97.60:45945
I20260620 03:06:17.322676 7663 heartbeater.cc:511] Master 127.0.97.62:45499 requested a full tablet report, sending...
I20260620 03:06:17.322764 7662 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:17.323244 7662 heartbeater.cc:511] Master 127.0.97.60:45945 requested a full tablet report, sending...
I20260620 03:06:17.323553 7664 heartbeater.cc:347] Connected to a master server at 127.0.97.61:44195
I20260620 03:06:17.323709 7664 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:17.325035 7336 ts_manager.cc:194] Registered new tserver with Master: b56f067b01d34ef1b8c339beaf862a84 (127.0.97.2:35015)
I20260620 03:06:17.325194 6546 ts_manager.cc:194] Re-registered known tserver with Master: b56f067b01d34ef1b8c339beaf862a84 (127.0.97.2:35015)
I20260620 03:06:17.326364 6546 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40595
I20260620 03:06:17.327086 7664 heartbeater.cc:511] Master 127.0.97.61:44195 requested a full tablet report, sending...
I20260620 03:06:17.328254 7148 ts_manager.cc:194] Registered new tserver with Master: b56f067b01d34ef1b8c339beaf862a84 (127.0.97.2:35015)
I20260620 03:06:17.408015 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 6874
I20260620 03:06:17.424345 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35005
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45499
--webserver_port=46791
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:45499,127.0.97.61:44195
--tserver_master_addrs=127.0.97.62:45499,127.0.97.61:44195,127.0.97.60:45945 with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:06:17.714221 7516 heartbeater.cc:503] Master 127.0.97.62:45499 was elected leader, sending a full tablet report...
W20260620 03:06:17.744081 7675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:17.744357 7675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:17.744477 7675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:17.751830 7675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:17.751966 7675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:17.752146 7675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:06:17.761758 7675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=46791
--tserver_master_addrs=127.0.97.62:45499,127.0.97.61:44195,127.0.97.60:45945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:17.763180 7675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:17.764463 7675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:17.777045 7681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:17.777132 7683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:17.777109 7680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:17.778231 7675 server_base.cc:1034] running on GCE node
I20260620 03:06:17.778765 7675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:17.779683 7675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:17.780876 7675 hybrid_clock.cc:648] HybridClock initialized: now 1781924777780825 us; error 44 us; skew 500 ppm
I20260620 03:06:17.781136 7675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:17 dist-test-slave-xv68 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:17.784256 7675 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 03:06:17.785681 7675 webserver.cc:460] Webserver started at http://127.0.97.3:46791/ using document root <none> and password file <none>
I20260620 03:06:17.786360 7675 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:17.786497 7675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:17.791644 7675 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:06:17.794162 7690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:17.795480 7675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:06:17.795608 7675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853"
format_stamp: "Formatted at 2026-06-20 03:06:14 on dist-test-slave-xv68"
server_key: "cba4bb4416eb6048f720d884a2d1ec37"
server_key_iv: "fec4a2794fdd0c86ba9c4dc6ee384df1"
server_key_version: "encryptionkey@0"
I20260620 03:06:17.796480 7675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:17.821074 7675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:17.825361 7675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:17.825714 7675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:17.826704 7675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:17.828035 7675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:17.828117 7675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:17.828184 7675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:17.828238 7675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:17.860585 7675 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35005
I20260620 03:06:17.860683 7802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35005 every 8 connection(s)
I20260620 03:06:17.862016 7675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:06:17.864468 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 7675
Jun 20 03:06:17 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 03:06:17 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 03:06:17 dist-test-slave-xv68 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 03:06:17.925588 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.866728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51985 (local address 127.0.97.61:44195)
0620 03:06:17.888569 (+ 21841us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:17.888578 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:17.889491 (+ 913us) server_negotiation.cc:407] Connection header received
0620 03:06:17.889563 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:17.889571 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:17.889672 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:17.889794 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:17.894263 (+ 4469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.895278 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.906294 (+ 11016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.906995 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.907122 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.916757 (+ 9635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:17.916765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:17.916770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:17.916825 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:17.921430 (+ 4605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.922649 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.922654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.922658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.922805 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.923465 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.923469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.923472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.923815 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:17.923938 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:17.924927 (+ 989us) server_negotiation.cc:299] Negotiation successful
0620 03:06:17.925169 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21681,"thread_start_us":21567,"threads_started":1}
I20260620 03:06:17.926229 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.892019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57978 (local address 127.0.97.61:44195)
0620 03:06:17.893236 (+ 1217us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:17.893242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:17.895079 (+ 1837us) server_negotiation.cc:407] Connection header received
0620 03:06:17.895329 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:17.895338 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:17.895413 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:17.895500 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:17.898818 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.901061 (+ 2243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.903678 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.904561 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.904680 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.909308 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:17.909315 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:17.909319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:17.909355 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:17.914450 (+ 5095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.923334 (+ 8884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.923338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.923341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.923575 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.924025 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.924029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.924031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.924313 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:17.924528 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:17.925647 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0620 03:06:17.925862 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":1689,"threads_started":1}
I20260620 03:06:17.928481 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.866759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44195 (local address 127.0.97.3:51985)
0620 03:06:17.889119 (+ 22360us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:17.889144 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:17.889264 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:17.890241 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:17.890253 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:17.892508 (+ 2255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:17.894104 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:17.894115 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.897910 (+ 3795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.897915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:17.906150 (+ 8235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:17.906170 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.907610 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.907620 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:17.907807 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.910308 (+ 2501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:17.910336 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:17.916100 (+ 5764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:17.921590 (+ 5490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:17.921601 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:17.921624 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:17.922321 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:17.922986 (+ 665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:17.922991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:17.922994 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:17.923343 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:17.924048 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:17.924056 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:17.924495 (+ 439us) client_negotiation.cc:769] Sending connection context
0620 03:06:17.926131 (+ 1636us) client_negotiation.cc:240] Negotiation successful
0620 03:06:17.927004 (+ 873us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22206,"mutex_wait_us":93,"thread_start_us":16813,"threads_started":1}
I20260620 03:06:17.930224 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.864954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51977 (local address 127.0.97.62:45499)
0620 03:06:17.882813 (+ 17859us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:17.882823 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:17.884295 (+ 1472us) server_negotiation.cc:407] Connection header received
0620 03:06:17.891428 (+ 7133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:17.891435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:17.891613 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:17.891722 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:17.893762 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.894958 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.897834 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.898444 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.898572 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.915589 (+ 17017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:17.915597 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:17.915602 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:17.915642 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:17.921245 (+ 5603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.927709 (+ 6464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.927714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.927717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.927862 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.928596 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.928604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.928608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.928940 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:17.929060 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:17.929613 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 03:06:17.929854 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17677,"thread_start_us":17856,"threads_started":1}
I20260620 03:06:17.930517 7805 heartbeater.cc:347] Connected to a master server at 127.0.97.61:44195
I20260620 03:06:17.930830 7805 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:17.931430 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.891555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.0.97.60:45945)
0620 03:06:17.908655 (+ 17100us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:17.908661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:17.908676 (+ 15us) server_negotiation.cc:407] Connection header received
0620 03:06:17.908717 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:17.908721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:17.908793 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:17.910843 (+ 2050us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:17.911058 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.912573 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.915207 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.916713 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.916832 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.920653 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:17.920659 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:17.920665 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:17.920703 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:17.925358 (+ 4655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.927523 (+ 2165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.927530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.927535 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.927701 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.928134 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.928138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.928141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.928492 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:17.930576 (+ 2084us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:17.930712 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 03:06:17.930969 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16979,"thread_start_us":13611,"threads_started":1}
I20260620 03:06:17.931615 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.865727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:51977)
0620 03:06:17.884035 (+ 18308us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:17.884086 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:17.887559 (+ 3473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:17.891922 (+ 4363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:17.891930 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:17.892508 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:17.893607 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:17.893635 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.895481 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.895487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:17.897700 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:17.897714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.905249 (+ 7535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.905256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:17.905577 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.908458 (+ 2881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:17.908498 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:17.915377 (+ 6879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:17.925268 (+ 9891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:17.925275 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:17.925281 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:17.925754 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:17.928251 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:17.928256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:17.928259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:17.928437 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:17.929148 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:17.929155 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:17.929322 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 03:06:17.930389 (+ 1067us) client_negotiation.cc:240] Negotiation successful
0620 03:06:17.930706 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17968,"mutex_wait_us":234,"spinlock_wait_cycles":17664,"thread_start_us":22891,"threads_started":1}
I20260620 03:06:17.931819 7805 heartbeater.cc:511] Master 127.0.97.61:44195 requested a full tablet report, sending...
I20260620 03:06:17.932927 7804 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45499
I20260620 03:06:17.933058 7804 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:17.933940 7804 heartbeater.cc:511] Master 127.0.97.62:45499 requested a full tablet report, sending...
I20260620 03:06:17.933983 7148 ts_manager.cc:194] Registered new tserver with Master: c4a1d8ef8d9b4cc391b7a657fb81a853 (127.0.97.3:35005)
I20260620 03:06:17.934902 6546 ts_manager.cc:194] Re-registered known tserver with Master: c4a1d8ef8d9b4cc391b7a657fb81a853 (127.0.97.3:35005)
I20260620 03:06:17.936012 6546 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51977
I20260620 03:06:17.936877 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.883982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45945 (local address 127.0.97.3:55057)
0620 03:06:17.913164 (+ 29182us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:17.913181 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:17.913325 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:17.913856 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:17.913860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:17.914458 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:17.914718 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:17.914728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.916096 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.916103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:17.917567 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:17.917583 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.918781 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.918788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:17.918988 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.919610 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:17.919638 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:17.926439 (+ 6801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:17.934155 (+ 7716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:17.934160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:17.934163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:17.934642 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:17.935063 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:17.935068 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:17.935071 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:17.935218 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:17.935828 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:17.935834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:17.935998 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 03:06:17.936182 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 03:06:17.936455 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29059,"thread_start_us":28895,"threads_started":1}
I20260620 03:06:17.936908 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.884486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55057 (local address 127.0.97.60:45945)
0620 03:06:17.905447 (+ 20961us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:17.905456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:17.913278 (+ 7822us) server_negotiation.cc:407] Connection header received
0620 03:06:17.913448 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:17.913456 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:17.913532 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:17.913638 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:17.914910 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.915964 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.917732 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.918399 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.918516 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.926667 (+ 8151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:17.926673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:17.926677 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:17.926720 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:17.933988 (+ 7268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.934778 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.934782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.934785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.934925 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.935372 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.935376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.935379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.935692 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:17.935822 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:17.936266 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 03:06:17.936534 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":20807,"thread_start_us":23938,"threads_started":1}
I20260620 03:06:17.938017 7803 heartbeater.cc:347] Connected to a master server at 127.0.97.60:45945
I20260620 03:06:17.938128 7803 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:17.938647 7803 heartbeater.cc:511] Master 127.0.97.60:45945 requested a full tablet report, sending...
I20260620 03:06:17.939765 7336 ts_manager.cc:194] Registered new tserver with Master: c4a1d8ef8d9b4cc391b7a657fb81a853 (127.0.97.3:35005)
I20260620 03:06:17.941766 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.899511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.0.97.62:45499)
0620 03:06:17.922758 (+ 23247us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:17.922765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:17.922784 (+ 19us) server_negotiation.cc:407] Connection header received
0620 03:06:17.922829 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:17.922832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:17.922887 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:17.923007 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:17.925499 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.926652 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.929493 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.930183 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.930308 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.935426 (+ 5118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:17.935434 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:17.935443 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:17.935474 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:17.939017 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.939768 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.939772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.939774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.939922 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:17.940329 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:17.940332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:17.940334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:17.940605 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:17.940770 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:17.941219 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 03:06:17.941375 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23091,"thread_start_us":22959,"threads_started":1}
I20260620 03:06:17.945557 6546 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:06:17.953020 6546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:06:17.985514 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.971158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33584 (local address 127.0.97.3:35005)
0620 03:06:17.977322 (+ 6164us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:17.977329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:17.977357 (+ 28us) server_negotiation.cc:407] Connection header received
0620 03:06:17.977445 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:17.977451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:17.977685 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:17.977866 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:17.978487 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.981162 (+ 2675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.983479 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.984527 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.984653 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.984850 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:17.985072 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 03:06:17.985218 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6015,"thread_start_us":5842,"threads_started":1}
I20260620 03:06:17.985536 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.970790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35005 (local address 127.0.0.1:33584)
0620 03:06:17.971454 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:17.971475 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:17.971599 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:17.977878 (+ 6279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:17.977885 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:17.977920 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:17.978320 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:17.978333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.981266 (+ 2933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.981271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:17.983234 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:17.983244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.984714 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.984721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:17.984875 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.984908 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 03:06:17.985025 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 03:06:17.985146 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":333,"threads_started":1}
I20260620 03:06:17.986382 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.971612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35015 (local address 127.0.0.1:57008)
0620 03:06:17.972432 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:17.972452 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:17.972626 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:17.976621 (+ 3995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:17.976629 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:17.976667 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:17.977112 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:17.977126 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.978928 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.978934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:17.981098 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:17.981114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.984444 (+ 3330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.984451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:17.985766 (+ 1315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.985797 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 03:06:17.985926 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 03:06:17.986049 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":4213,"threads_started":1}
I20260620 03:06:17.986970 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.972244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.0.97.1:38365)
0620 03:06:17.979458 (+ 7214us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:17.979466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:17.979490 (+ 24us) server_negotiation.cc:407] Connection header received
0620 03:06:17.979584 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:17.979591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:17.979790 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:17.979969 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:17.980908 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.982407 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.984838 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.985789 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.985901 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.986078 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:17.986486 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 03:06:17.986691 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7029,"thread_start_us":6907,"threads_started":1}
I20260620 03:06:17.987004 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.972120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.0.97.2:35015)
0620 03:06:17.975089 (+ 2969us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:17.975101 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:17.975134 (+ 33us) server_negotiation.cc:407] Connection header received
0620 03:06:17.975244 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:17.975254 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:17.976380 (+ 1126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:17.976630 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:17.977301 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.978754 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.982622 (+ 3868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.984239 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.984365 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.984570 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:17.986598 (+ 2028us) server_negotiation.cc:299] Negotiation successful
0620 03:06:17.986739 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2813,"thread_start_us":2736,"threads_started":1}
I20260620 03:06:17.987023 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:17.971771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38365 (local address 127.0.0.1:34218)
0620 03:06:17.976072 (+ 4301us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:17.976093 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:17.976223 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:17.979983 (+ 3760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:17.979986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:17.980005 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:17.980714 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:17.980723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.982533 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.982538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:17.984695 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:17.984706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:17.985958 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:17.985964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:17.986144 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:17.986182 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 03:06:17.986262 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 03:06:17.986377 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4170,"thread_start_us":582,"threads_started":1}
I20260620 03:06:17.989776 7738 tablet_service.cc:1467] Processing CreateTablet for tablet a5c1f9db7fc942169acf93673c2af87e (DEFAULT_TABLE table=test-table [id=796c4dc78a9346bfa19efba0e110a9a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:17.990919 7738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5c1f9db7fc942169acf93673c2af87e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:17.990582 7597 tablet_service.cc:1467] Processing CreateTablet for tablet a5c1f9db7fc942169acf93673c2af87e (DEFAULT_TABLE table=test-table [id=796c4dc78a9346bfa19efba0e110a9a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:17.991375 7450 tablet_service.cc:1467] Processing CreateTablet for tablet a5c1f9db7fc942169acf93673c2af87e (DEFAULT_TABLE table=test-table [id=796c4dc78a9346bfa19efba0e110a9a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:17.991894 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5c1f9db7fc942169acf93673c2af87e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:17.993395 7450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5c1f9db7fc942169acf93673c2af87e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:18.009075 7837 tablet_bootstrap.cc:492] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853: Bootstrap starting.
I20260620 03:06:18.013471 7837 tablet_bootstrap.cc:654] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:18.015017 7837 log.cc:824] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:18.016076 7838 tablet_bootstrap.cc:492] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84: Bootstrap starting.
I20260620 03:06:18.019126 7838 tablet_bootstrap.cc:654] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:18.020020 7839 tablet_bootstrap.cc:492] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d: Bootstrap starting.
I20260620 03:06:18.020411 7838 log.cc:824] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:18.023254 7838 tablet_bootstrap.cc:492] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84: No bootstrap required, opened a new log
I20260620 03:06:18.023535 7838 ts_tablet_manager.cc:1397] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 03:06:18.023896 7839 tablet_bootstrap.cc:654] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:18.025182 7839 log.cc:824] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:18.027659 7839 tablet_bootstrap.cc:492] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d: No bootstrap required, opened a new log
I20260620 03:06:18.027915 7839 ts_tablet_manager.cc:1397] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 03:06:18.028903 7837 tablet_bootstrap.cc:492] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853: No bootstrap required, opened a new log
I20260620 03:06:18.029170 7837 ts_tablet_manager.cc:1397] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20260620 03:06:18.034523 7838 raft_consensus.cc:348] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } } peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } }
I20260620 03:06:18.034970 7838 raft_consensus.cc:374] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:18.035068 7838 raft_consensus.cc:729] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b56f067b01d34ef1b8c339beaf862a84, State: Initialized, Role: FOLLOWER
I20260620 03:06:18.035275 7839 raft_consensus.cc:348] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } } peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } }
I20260620 03:06:18.035708 7839 raft_consensus.cc:374] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:18.035805 7839 raft_consensus.cc:729] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63870f6d79a144b7bd7696c4c4c47b4d, State: Initialized, Role: FOLLOWER
I20260620 03:06:18.035702 7838 consensus_queue.cc:260] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } } peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } }
I20260620 03:06:18.036330 7839 consensus_queue.cc:260] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } } peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } }
I20260620 03:06:18.040179 7663 heartbeater.cc:503] Master 127.0.97.62:45499 was elected leader, sending a full tablet report...
I20260620 03:06:18.040760 7838 ts_tablet_manager.cc:1428] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260620 03:06:18.042098 7837 raft_consensus.cc:348] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } } peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } }
I20260620 03:06:18.042495 7837 raft_consensus.cc:374] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:18.042595 7837 raft_consensus.cc:729] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4a1d8ef8d9b4cc391b7a657fb81a853, State: Initialized, Role: FOLLOWER
I20260620 03:06:18.043118 7837 consensus_queue.cc:260] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } } peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } }
I20260620 03:06:18.049113 7839 ts_tablet_manager.cc:1428] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d: Time spent starting tablet: real 0.021s user 0.009s sys 0.012s
I20260620 03:06:18.051573 7837 ts_tablet_manager.cc:1428] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853: Time spent starting tablet: real 0.022s user 0.015s sys 0.007s
I20260620 03:06:18.053778 7804 heartbeater.cc:503] Master 127.0.97.62:45499 was elected leader, sending a full tablet report...
W20260620 03:06:18.120604 7806 tablet.cc:2369] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:06:18.131992 7844 raft_consensus.cc:482] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:06:18.132256 7844 raft_consensus.cc:504] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } } peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } }
I20260620 03:06:18.134086 7844 leader_election.cc:288] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4a1d8ef8d9b4cc391b7a657fb81a853 (127.0.97.3:35005), b56f067b01d34ef1b8c339beaf862a84 (127.0.97.2:35015)
I20260620 03:06:18.141484 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:18.134419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43105 (local address 127.0.97.3:35005)
0620 03:06:18.134661 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:18.134671 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:18.135260 (+ 589us) server_negotiation.cc:407] Connection header received
0620 03:06:18.135514 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:18.135520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:18.135604 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:18.135686 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:18.136263 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:18.137014 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:18.139750 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:18.140513 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:18.140612 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:18.140749 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:18.141103 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 03:06:18.141244 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 03:06:18.141635 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:18.134534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55167 (local address 127.0.97.2:35015)
0620 03:06:18.134859 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:18.134865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:18.135460 (+ 595us) server_negotiation.cc:407] Connection header received
0620 03:06:18.135635 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:18.135639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:18.135738 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:18.135832 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:18.136416 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:18.137144 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:18.139822 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:18.140664 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:18.140786 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:18.140882 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:18.141233 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 03:06:18.141371 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260620 03:06:18.141788 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:18.134538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35015 (local address 127.0.97.1:55167)
0620 03:06:18.135360 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:18.135373 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:18.135511 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:18.135857 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:18.135861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:18.135883 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:18.136223 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:18.136235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:18.137281 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:18.137285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:18.139648 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:18.139682 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:18.140929 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:18.140934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:18.141069 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:18.141100 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 03:06:18.141197 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 03:06:18.141308 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":504,"threads_started":1}
I20260620 03:06:18.142086 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:18.134226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35005 (local address 127.0.97.1:43105)
0620 03:06:18.135126 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:18.135146 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:18.135359 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:18.135736 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:18.135741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:18.135766 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:18.136120 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:18.136129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:18.137159 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:18.137164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:18.139580 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:18.139600 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:18.140764 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:18.140771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:18.140957 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:18.140982 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 03:06:18.141056 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 03:06:18.141144 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":49,"thread_start_us":551,"threads_started":1}
I20260620 03:06:18.142724 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c1f9db7fc942169acf93673c2af87e" candidate_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b56f067b01d34ef1b8c339beaf862a84" is_pre_election: true
I20260620 03:06:18.143114 7758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c1f9db7fc942169acf93673c2af87e" candidate_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" is_pre_election: true
I20260620 03:06:18.143325 7617 raft_consensus.cc:2454] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63870f6d79a144b7bd7696c4c4c47b4d in term 0.
I20260620 03:06:18.143664 7758 raft_consensus.cc:2454] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63870f6d79a144b7bd7696c4c4c47b4d in term 0.
I20260620 03:06:18.144003 7404 leader_election.cc:302] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63870f6d79a144b7bd7696c4c4c47b4d, b56f067b01d34ef1b8c339beaf862a84; no voters:
I20260620 03:06:18.144413 7844 raft_consensus.cc:2781] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:06:18.144572 7844 raft_consensus.cc:482] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:06:18.144690 7844 raft_consensus.cc:3037] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:18.146836 7844 raft_consensus.cc:504] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } } peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } }
I20260620 03:06:18.147500 7844 leader_election.cc:288] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [CANDIDATE]: Term 1 election: Requested vote from peers c4a1d8ef8d9b4cc391b7a657fb81a853 (127.0.97.3:35005), b56f067b01d34ef1b8c339beaf862a84 (127.0.97.2:35015)
I20260620 03:06:18.147866 7758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c1f9db7fc942169acf93673c2af87e" candidate_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853"
I20260620 03:06:18.147905 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c1f9db7fc942169acf93673c2af87e" candidate_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b56f067b01d34ef1b8c339beaf862a84"
I20260620 03:06:18.148066 7617 raft_consensus.cc:3037] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:18.148063 7758 raft_consensus.cc:3037] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:18.151588 7758 raft_consensus.cc:2454] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63870f6d79a144b7bd7696c4c4c47b4d in term 1.
I20260620 03:06:18.151588 7617 raft_consensus.cc:2454] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63870f6d79a144b7bd7696c4c4c47b4d in term 1.
I20260620 03:06:18.151937 7404 leader_election.cc:302] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63870f6d79a144b7bd7696c4c4c47b4d, b56f067b01d34ef1b8c339beaf862a84; no voters:
I20260620 03:06:18.152190 7844 raft_consensus.cc:2781] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:18.152627 7844 raft_consensus.cc:686] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [term 1 LEADER]: Becoming Leader. State: Replica: 63870f6d79a144b7bd7696c4c4c47b4d, State: Running, Role: LEADER
I20260620 03:06:18.153584 7844 consensus_queue.cc:237] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } } peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } }
I20260620 03:06:18.158922 6546 catalog_manager.cc:5495] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 63870f6d79a144b7bd7696c4c4c47b4d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } health_report { overall_health: HEALTHY } } }
W20260620 03:06:18.182183 7518 tablet.cc:2369] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:06:18.221690 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:18.216437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.0.97.1:38365)
0620 03:06:18.216646 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:18.216652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:18.216670 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:06:18.216847 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:18.216854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:18.216932 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:18.217095 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:06:18.217429 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:18.218122 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:18.219510 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:18.220048 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:18.220150 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:18.220580 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:06:18.221095 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:06:18.221240 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:18.221341 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 03:06:18.221429 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 03:06:18.229561 7758 raft_consensus.cc:1261] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853 [term 1 FOLLOWER]: Refusing update from remote peer 63870f6d79a144b7bd7696c4c4c47b4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:06:18.229561 7617 raft_consensus.cc:1261] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84 [term 1 FOLLOWER]: Refusing update from remote peer 63870f6d79a144b7bd7696c4c4c47b4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:06:18.230633 7848 consensus_queue.cc:1035] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:18.230989 7844 consensus_queue.cc:1035] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:18.245745 7852 mvcc.cc:205] Tried to move back new op lower bound from 7298763891617705984 to 7298763891320467456. Current Snapshot: MvccSnapshot[applied={T|T < 7298763891617705984}]
I20260620 03:06:18.253474 7853 mvcc.cc:205] Tried to move back new op lower bound from 7298763891617705984 to 7298763891320467456. Current Snapshot: MvccSnapshot[applied={T|T < 7298763891617705984}]
I20260620 03:06:18.268025 6546 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:06:18.268265 6546 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:06:18.276988 6546 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85a88abe479747acb7a7878d64c9d24c: Sending DeleteTablet for 3 replicas of tablet a5c1f9db7fc942169acf93673c2af87e
I20260620 03:06:18.278090 7738 tablet_service.cc:1514] Processing DeleteTablet for tablet a5c1f9db7fc942169acf93673c2af87e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33584
I20260620 03:06:18.278537 7450 tablet_service.cc:1514] Processing DeleteTablet for tablet a5c1f9db7fc942169acf93673c2af87e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34218
I20260620 03:06:18.278903 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet a5c1f9db7fc942169acf93673c2af87e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57008
I20260620 03:06:18.291199 7862 tablet_replica.cc:330] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853: stopping tablet replica
I20260620 03:06:18.291695 7862 raft_consensus.cc:2229] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:06:18.292245 7862 raft_consensus.cc:2258] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:18.296401 7862 ts_tablet_manager.cc:1905] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:18.297569 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 6514
I20260620 03:06:18.298858 6576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:18.299588 7864 tablet_replica.cc:330] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84: stopping tablet replica
I20260620 03:06:18.303591 7862 ts_tablet_manager.cc:1918] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:06:18.303746 7862 log.cc:1196] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/a5c1f9db7fc942169acf93673c2af87e
I20260620 03:06:18.304260 7862 ts_tablet_manager.cc:1939] T a5c1f9db7fc942169acf93673c2af87e P c4a1d8ef8d9b4cc391b7a657fb81a853: Deleting consensus metadata
I20260620 03:06:18.306488 7864 raft_consensus.cc:2229] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:06:18.306979 7864 raft_consensus.cc:2258] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:18.307777 7863 tablet_replica.cc:330] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d: stopping tablet replica
I20260620 03:06:18.308161 7863 raft_consensus.cc:2229] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:06:18.308856 7863 raft_consensus.cc:2258] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:18.310756 7864 ts_tablet_manager.cc:1905] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:18.317005 7863 ts_tablet_manager.cc:1905] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:18.320785 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:18.281784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57992 (local address 127.0.97.61:44195)
0620 03:06:18.281964 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:18.282027 (+ 63us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:18.299084 (+ 17057us) server_negotiation.cc:407] Connection header received
0620 03:06:18.299190 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:18.299197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:18.299319 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:18.299417 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:18.303267 (+ 3850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:18.304745 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:18.307666 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:18.308602 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:18.308727 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:18.313554 (+ 4827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:18.313561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:18.313566 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:18.313602 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:18.317564 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:18.318384 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:18.318390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:18.318394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:18.318550 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:18.318971 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:18.318976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:18.318979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:18.319340 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:18.319563 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:18.320182 (+ 619us) server_negotiation.cc:299] Negotiation successful
0620 03:06:18.320393 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 03:06:18.326315 7864 ts_tablet_manager.cc:1918] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:06:18.326483 7864 log.cc:1196] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/a5c1f9db7fc942169acf93673c2af87e
I20260620 03:06:18.327044 7864 ts_tablet_manager.cc:1939] T a5c1f9db7fc942169acf93673c2af87e P b56f067b01d34ef1b8c339beaf862a84: Deleting consensus metadata
I20260620 03:06:18.329718 7863 ts_tablet_manager.cc:1918] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:06:18.329882 7863 log.cc:1196] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/a5c1f9db7fc942169acf93673c2af87e
I20260620 03:06:18.330468 7863 ts_tablet_manager.cc:1939] T a5c1f9db7fc942169acf93673c2af87e P 63870f6d79a144b7bd7696c4c4c47b4d: Deleting consensus metadata
I20260620 03:06:18.340901 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:18.296069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57092 (local address 127.0.97.60:45945)
0620 03:06:18.296315 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:18.296323 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:18.323284 (+ 26961us) server_negotiation.cc:407] Connection header received
0620 03:06:18.323592 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:18.323597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:18.323677 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:18.323745 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:18.326258 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:18.327362 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:18.329467 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:18.330233 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:18.330344 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:18.334133 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:18.334140 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:18.334146 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:18.334187 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:18.338041 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:18.338785 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:18.338790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:18.338794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:18.338947 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:18.339405 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:18.339410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:18.339414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:18.339762 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:18.340027 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:18.340339 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 03:06:18.340559 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 03:06:18.440366 6531 catalog_manager.cc:4841] TS b56f067b01d34ef1b8c339beaf862a84 (127.0.97.2:35015): tablet a5c1f9db7fc942169acf93673c2af87e (table test-table [id=796c4dc78a9346bfa19efba0e110a9a6]) successfully deleted
I20260620 03:06:18.440706 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 6514
W20260620 03:06:18.462021 7403 connection.cc:535] server connection from 127.0.0.1:34218 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 03:06:18.494856 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:18.466593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57996 (local address 127.0.97.61:44195)
0620 03:06:18.466797 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:18.466803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:18.477345 (+ 10542us) server_negotiation.cc:407] Connection header received
0620 03:06:18.477818 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:18.477824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:18.477908 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:18.477982 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:18.480856 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:18.481954 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:18.484187 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:18.484811 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:18.484908 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:18.488605 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:18.488614 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:18.488621 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:18.488665 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:18.491504 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:18.492736 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:18.492741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:18.492745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:18.492891 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:18.493298 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:18.493303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:18.493306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:18.493670 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:18.493936 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:18.494255 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 03:06:18.494495 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 03:06:18.496968 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:18.467843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57102 (local address 127.0.97.60:45945)
0620 03:06:18.470777 (+ 2934us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:18.470783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:18.480692 (+ 9909us) server_negotiation.cc:407] Connection header received
0620 03:06:18.480843 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:18.480851 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:18.480960 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:18.481086 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:18.484137 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:18.485639 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:18.487431 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:18.487868 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:18.488010 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:18.490884 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:18.490892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:18.490900 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:18.490941 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:18.494176 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:18.494958 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:18.494963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:18.494967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:18.495097 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:18.495524 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:18.495529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:18.495533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:18.495841 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:18.496105 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:18.496426 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 03:06:18.496628 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2792}
I20260620 03:06:19.262085 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:19.261260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:41859)
0620 03:06:19.261704 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:19.261889 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":205,"threads_started":1}
W20260620 03:06:19.262651 7516 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45499 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
I20260620 03:06:19.262796 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:19.262024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:55187)
0620 03:06:19.262496 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:19.262620 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":242,"threads_started":1}
W20260620 03:06:19.263815 7663 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45499 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
I20260620 03:06:19.264732 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:19.263447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:50165)
0620 03:06:19.264350 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:19.264478 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":698,"threads_started":1}
W20260620 03:06:19.265077 7804 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45499 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
I20260620 03:06:19.776392 7877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a88abe479747acb7a7878d64c9d24c)
I20260620 03:06:19.776710 7877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } }
I20260620 03:06:19.776851 7878 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a88abe479747acb7a7878d64c9d24c)
I20260620 03:06:19.777139 7878 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } }
I20260620 03:06:19.778160 7877 leader_election.cc:288] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a88abe479747acb7a7878d64c9d24c (127.0.97.62:45499), d7ed128efced40ed8d9d0b9fd4ada0f5 (127.0.97.60:45945)
I20260620 03:06:19.778340 7878 leader_election.cc:288] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a88abe479747acb7a7878d64c9d24c (127.0.97.62:45499), 5f373e2277904013a4c4acee64771191 (127.0.97.61:44195)
I20260620 03:06:19.779801 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:19.778206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:47362)
0620 03:06:19.779404 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:19.779578 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1054,"threads_started":1}
I20260620 03:06:19.780954 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:19.778797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:47368)
0620 03:06:19.780315 (+ 1518us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:19.780563 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1013,"thread_start_us":1410,"threads_started":1}
W20260620 03:06:19.781822 7135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
W20260620 03:06:19.781915 7323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
I20260620 03:06:19.785259 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:19.784645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:47370)
0620 03:06:19.784830 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:19.784966 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 03:06:19.785511 7135 leader_election.cc:334] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 85a88abe479747acb7a7878d64c9d24c (127.0.97.62:45499): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
I20260620 03:06:19.787807 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:19.787409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:47382)
0620 03:06:19.787571 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:19.787670 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 03:06:19.788198 7323 leader_election.cc:334] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 85a88abe479747acb7a7878d64c9d24c (127.0.97.62:45499): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
I20260620 03:06:19.788993 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:19.779127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57998 (local address 127.0.97.61:44195)
0620 03:06:19.780270 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:19.780275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:19.780290 (+ 15us) server_negotiation.cc:407] Connection header received
0620 03:06:19.780341 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:19.780346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:19.780410 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:19.781294 (+ 884us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:19.781582 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:19.783218 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:19.786564 (+ 3346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:19.787742 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:19.787859 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:19.787978 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:19.788569 (+ 591us) server_negotiation.cc:299] Negotiation successful
0620 03:06:19.788723 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":1047,"thread_start_us":868,"threads_started":1}
I20260620 03:06:19.789496 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:19.778586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44195 (local address 127.0.0.1:57998)
0620 03:06:19.779639 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:19.779659 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:19.779797 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:19.780562 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:19.780568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:19.780606 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:19.781093 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:19.781110 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:19.783508 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:19.783521 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:19.786397 (+ 2876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:19.786412 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:19.788098 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:19.788110 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:19.788342 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:19.788398 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 03:06:19.789165 (+ 767us) client_negotiation.cc:240] Negotiation successful
0620 03:06:19.789267 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"thread_start_us":817,"threads_started":1}
I20260620 03:06:19.790138 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:19.780424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57106 (local address 127.0.97.60:45945)
0620 03:06:19.781948 (+ 1524us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:19.781953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:19.781971 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:06:19.782178 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:19.782184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:19.782274 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:19.782402 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:19.783368 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:19.785751 (+ 2383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:19.787896 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:19.789156 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:19.789267 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:19.789381 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:19.789732 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 03:06:19.789858 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1384,"thread_start_us":1262,"threads_started":1}
I20260620 03:06:19.790189 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:19.778379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45945 (local address 127.0.0.1:57106)
0620 03:06:19.779999 (+ 1620us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:19.780019 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:19.780139 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:19.782686 (+ 2547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:19.782694 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:19.782737 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:19.783104 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:19.783120 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:19.785898 (+ 2778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:19.785906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:19.787717 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:19.787731 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:19.789377 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:19.789383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:19.789543 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:19.789564 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 03:06:19.789672 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 03:06:19.789810 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1507,"mutex_wait_us":42,"thread_start_us":743,"threads_started":1}
I20260620 03:06:19.790200 7158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5f373e2277904013a4c4acee64771191" is_pre_election: true
I20260620 03:06:19.790678 7158 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ed128efced40ed8d9d0b9fd4ada0f5 in term 1.
I20260620 03:06:19.790803 7345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f373e2277904013a4c4acee64771191" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" is_pre_election: true
I20260620 03:06:19.791127 7345 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f373e2277904013a4c4acee64771191 in term 1.
I20260620 03:06:19.791167 7322 leader_election.cc:302] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f373e2277904013a4c4acee64771191, d7ed128efced40ed8d9d0b9fd4ada0f5; no voters: 85a88abe479747acb7a7878d64c9d24c
I20260620 03:06:19.791601 7878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 03:06:19.791721 7878 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a88abe479747acb7a7878d64c9d24c)
I20260620 03:06:19.791662 7132 leader_election.cc:302] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f373e2277904013a4c4acee64771191, d7ed128efced40ed8d9d0b9fd4ada0f5; no voters: 85a88abe479747acb7a7878d64c9d24c
I20260620 03:06:19.791821 7878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 1 FOLLOWER]: Advancing to term 2
I20260620 03:06:19.792035 7877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 03:06:19.792151 7877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a88abe479747acb7a7878d64c9d24c)
I20260620 03:06:19.792246 7877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 1 FOLLOWER]: Advancing to term 2
I20260620 03:06:19.794901 7878 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } }
I20260620 03:06:19.795104 7877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } }
I20260620 03:06:19.795646 7878 leader_election.cc:288] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [CANDIDATE]: Term 2 election: Requested vote from peers 85a88abe479747acb7a7878d64c9d24c (127.0.97.62:45499), 5f373e2277904013a4c4acee64771191 (127.0.97.61:44195)
I20260620 03:06:19.795773 7877 leader_election.cc:288] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [CANDIDATE]: Term 2 election: Requested vote from peers 85a88abe479747acb7a7878d64c9d24c (127.0.97.62:45499), d7ed128efced40ed8d9d0b9fd4ada0f5 (127.0.97.60:45945)
I20260620 03:06:19.796136 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:19.795772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:47392)
0620 03:06:19.795907 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:19.795995 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 03:06:19.796241 7158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5f373e2277904013a4c4acee64771191"
I20260620 03:06:19.796473 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:19.796078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:47398)
0620 03:06:19.796248 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:19.796327 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 03:06:19.796494 7158 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d7ed128efced40ed8d9d0b9fd4ada0f5 in current term 2: Already voted for candidate 5f373e2277904013a4c4acee64771191 in this term.
I20260620 03:06:19.796690 7345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f373e2277904013a4c4acee64771191" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5"
I20260620 03:06:19.797005 7345 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f373e2277904013a4c4acee64771191 in current term 2: Already voted for candidate d7ed128efced40ed8d9d0b9fd4ada0f5 in this term.
I20260620 03:06:19.797466 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:19.797104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:47404)
0620 03:06:19.797223 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:19.797345 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 03:06:19.797602 7135 leader_election.cc:334] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 85a88abe479747acb7a7878d64c9d24c (127.0.97.62:45499): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
I20260620 03:06:19.797676 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:19.797156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:47406)
0620 03:06:19.797445 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:19.797534 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260620 03:06:19.797751 7135 leader_election.cc:302] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f373e2277904013a4c4acee64771191; no voters: 85a88abe479747acb7a7878d64c9d24c, d7ed128efced40ed8d9d0b9fd4ada0f5
W20260620 03:06:19.797847 7323 leader_election.cc:334] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 85a88abe479747acb7a7878d64c9d24c (127.0.97.62:45499): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
I20260620 03:06:19.797961 7323 leader_election.cc:302] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7ed128efced40ed8d9d0b9fd4ada0f5; no voters: 5f373e2277904013a4c4acee64771191, 85a88abe479747acb7a7878d64c9d24c
I20260620 03:06:19.798013 7877 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 03:06:19.798226 7878 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 03:06:20.264207 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:20.263256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:57311)
0620 03:06:20.263853 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:20.263965 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":307,"threads_started":1}
I20260620 03:06:20.265302 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:20.264436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:44833)
0620 03:06:20.264947 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:20.265036 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":303,"threads_started":1}
I20260620 03:06:20.266362 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:20.265619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:42037)
0620 03:06:20.266059 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:20.266181 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":242,"threads_started":1}
I20260620 03:06:21.265852 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.265066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:43643)
0620 03:06:21.265525 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:21.265619 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":202,"threads_started":1}
I20260620 03:06:21.267264 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.266363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:37845)
0620 03:06:21.266998 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:21.267088 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":253,"threads_started":1}
I20260620 03:06:21.268165 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.267062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:37715)
0620 03:06:21.267904 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:21.267995 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":662,"threads_started":1}
I20260620 03:06:21.666249 7893 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:06:21.666429 7893 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } }
I20260620 03:06:21.667001 7893 leader_election.cc:288] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a88abe479747acb7a7878d64c9d24c (127.0.97.62:45499), 5f373e2277904013a4c4acee64771191 (127.0.97.61:44195)
I20260620 03:06:21.667640 7158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5f373e2277904013a4c4acee64771191" is_pre_election: true
I20260620 03:06:21.667886 7158 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ed128efced40ed8d9d0b9fd4ada0f5 in term 2.
I20260620 03:06:21.668076 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.667229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:47682)
0620 03:06:21.667743 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:21.667853 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":279,"threads_started":1}
I20260620 03:06:21.668361 7322 leader_election.cc:302] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f373e2277904013a4c4acee64771191, d7ed128efced40ed8d9d0b9fd4ada0f5; no voters:
I20260620 03:06:21.668644 7893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 03:06:21.668815 7893 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:06:21.668939 7893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 2 FOLLOWER]: Advancing to term 3
I20260620 03:06:21.669716 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.669333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:47698)
0620 03:06:21.669494 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:21.669571 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 03:06:21.669921 7323 leader_election.cc:334] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 85a88abe479747acb7a7878d64c9d24c (127.0.97.62:45499): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
I20260620 03:06:21.671255 7893 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } }
I20260620 03:06:21.671865 7893 leader_election.cc:288] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [CANDIDATE]: Term 3 election: Requested vote from peers 85a88abe479747acb7a7878d64c9d24c (127.0.97.62:45499), 5f373e2277904013a4c4acee64771191 (127.0.97.61:44195)
I20260620 03:06:21.672299 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.671968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:47712)
0620 03:06:21.672085 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:21.672174 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 03:06:21.672307 7158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5f373e2277904013a4c4acee64771191"
I20260620 03:06:21.672554 7158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 2 FOLLOWER]: Advancing to term 3
I20260620 03:06:21.673077 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.672794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:47714)
0620 03:06:21.672922 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:21.672989 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 03:06:21.673213 7323 leader_election.cc:334] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 85a88abe479747acb7a7878d64c9d24c (127.0.97.62:45499): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
I20260620 03:06:21.674624 7158 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ed128efced40ed8d9d0b9fd4ada0f5 in term 3.
I20260620 03:06:21.675015 7322 leader_election.cc:302] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f373e2277904013a4c4acee64771191, d7ed128efced40ed8d9d0b9fd4ada0f5; no voters: 85a88abe479747acb7a7878d64c9d24c
I20260620 03:06:21.675259 7893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 03:06:21.675756 7893 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [term 3 LEADER]: Becoming Leader. State: Replica: d7ed128efced40ed8d9d0b9fd4ada0f5, State: Running, Role: LEADER
I20260620 03:06:21.676131 7893 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } }
I20260620 03:06:21.679015 7896 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7ed128efced40ed8d9d0b9fd4ada0f5. Latest consensus state: current_term: 3 leader_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } } }
I20260620 03:06:21.679270 7896 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:21.679906 7898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:06:21.683283 7898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:06:21.684158 7898 catalog_manager.cc:1259] Loaded cluster ID: eec7f634163546948ee74189cc31cdbb
I20260620 03:06:21.684222 7898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:06:21.685246 7898 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:06:21.685942 7898 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: Loaded TSK: 0
I20260620 03:06:21.686692 7898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:06:21.767216 7335 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:06:21.774101 7335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:06:21.778353 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.777924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:47732)
0620 03:06:21.778135 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:21.778209 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 03:06:21.778378 7158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [term 3 FOLLOWER]: Refusing update from remote peer d7ed128efced40ed8d9d0b9fd4ada0f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 03:06:21.779084 7896 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 03:06:21.779452 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.779028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:47746)
0620 03:06:21.779182 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:21.779261 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 03:06:21.779655 7323 consensus_peers.cc:487] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 -> Peer 85a88abe479747acb7a7878d64c9d24c (127.0.97.62:45499): Couldn't send request to peer 85a88abe479747acb7a7878d64c9d24c. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 03:06:21.781507 7900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7ed128efced40ed8d9d0b9fd4ada0f5. Latest consensus state: current_term: 3 leader_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } } }
I20260620 03:06:21.781745 7900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:06:21.782846 7899 mvcc.cc:205] Tried to move back new op lower bound from 7298763906157338624 to 7298763905748819968. Current Snapshot: MvccSnapshot[applied={T|T < 7298763906157338624}]
I20260620 03:06:21.783691 7896 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } } }
I20260620 03:06:21.783983 7896 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:21.784309 7900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85a88abe479747acb7a7878d64c9d24c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } peers { permanent_uuid: "5f373e2277904013a4c4acee64771191" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44195 } attrs { promote: false } } peers { permanent_uuid: "d7ed128efced40ed8d9d0b9fd4ada0f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45945 } attrs { promote: false } } }
I20260620 03:06:21.784559 7900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f373e2277904013a4c4acee64771191 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:06:21.799327 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.792678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57018 (local address 127.0.97.2:35015)
0620 03:06:21.793551 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:21.793556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:21.793573 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:06:21.793649 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:21.793657 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:21.793729 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:21.794330 (+ 601us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:21.794485 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:21.795481 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:21.797614 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:21.798371 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:21.798516 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:21.798624 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:21.798853 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 03:06:21.798978 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":914,"threads_started":1}
I20260620 03:06:21.799383 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.792258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35015 (local address 127.0.0.1:57018)
0620 03:06:21.792874 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:21.792891 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:21.793000 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:21.793874 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:21.793881 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:21.793908 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:21.794187 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:21.794197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:21.795607 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:21.795613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:21.797473 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:21.797484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:21.798533 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:21.798538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:21.798704 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:21.798727 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 03:06:21.798829 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 03:06:21.798954 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":22,"thread_start_us":544,"threads_started":1}
I20260620 03:06:21.800310 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 5b0f5f67f87d4fa2b5702455cfe58a2f (DEFAULT_TABLE table=test-table [id=3622a457b9164cd1a25a0fbc9bfe981e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:21.800948 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b0f5f67f87d4fa2b5702455cfe58a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:21.801875 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.791969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.0.97.1:38365)
0620 03:06:21.793061 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:21.793067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:21.793085 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:06:21.793163 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:21.793168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:21.793244 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:21.793923 (+ 679us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:21.794094 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:21.795217 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:21.799665 (+ 4448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:21.800873 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:21.801023 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:21.801133 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:21.801476 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 03:06:21.801603 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":962,"thread_start_us":1219,"threads_started":1}
I20260620 03:06:21.802001 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.791870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38365 (local address 127.0.0.1:34242)
0620 03:06:21.792007 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:21.792027 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:21.792157 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:21.793421 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:21.793425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:21.793453 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:21.793783 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:21.793799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:21.795394 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:21.795402 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:21.798317 (+ 2915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:21.798335 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:21.801075 (+ 2740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:21.801081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:21.801333 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:21.801357 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 03:06:21.801452 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 03:06:21.801630 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":27}
I20260620 03:06:21.802603 7450 tablet_service.cc:1467] Processing CreateTablet for tablet 5b0f5f67f87d4fa2b5702455cfe58a2f (DEFAULT_TABLE table=test-table [id=3622a457b9164cd1a25a0fbc9bfe981e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:21.802914 7450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b0f5f67f87d4fa2b5702455cfe58a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:21.808116 7911 tablet_bootstrap.cc:492] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84: Bootstrap starting.
I20260620 03:06:21.809837 7911 tablet_bootstrap.cc:654] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:21.811764 7911 tablet_bootstrap.cc:492] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84: No bootstrap required, opened a new log
I20260620 03:06:21.811906 7911 ts_tablet_manager.cc:1397] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 03:06:21.812278 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.793125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35005 (local address 127.0.0.1:33588)
0620 03:06:21.793632 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:21.793653 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:21.793772 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:21.805047 (+ 11275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:21.805052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:21.805077 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:21.805397 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:21.805406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:21.806836 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:21.806843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:21.809053 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:21.809071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:21.811605 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:21.811609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:21.811721 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:21.811742 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 03:06:21.811839 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 03:06:21.811960 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":545,"threads_started":1}
I20260620 03:06:21.812285 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.793483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.0.97.3:35005)
0620 03:06:21.804586 (+ 11103us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:21.804592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:21.804611 (+ 19us) server_negotiation.cc:407] Connection header received
0620 03:06:21.804703 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:21.804709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:21.804780 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:21.805015 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:21.805591 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:21.806701 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:21.809659 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:21.811155 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:21.811228 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:21.811350 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:21.811866 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 03:06:21.812000 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10974,"thread_start_us":11378,"threads_started":1}
I20260620 03:06:21.812633 7911 raft_consensus.cc:348] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } } peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260620 03:06:21.812839 7911 raft_consensus.cc:374] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:21.812929 7911 raft_consensus.cc:729] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b56f067b01d34ef1b8c339beaf862a84, State: Initialized, Role: FOLLOWER
I20260620 03:06:21.813067 7738 tablet_service.cc:1467] Processing CreateTablet for tablet 5b0f5f67f87d4fa2b5702455cfe58a2f (DEFAULT_TABLE table=test-table [id=3622a457b9164cd1a25a0fbc9bfe981e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:21.813146 7911 consensus_queue.cc:260] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } } peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260620 03:06:21.813504 7738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b0f5f67f87d4fa2b5702455cfe58a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:21.813953 7911 ts_tablet_manager.cc:1428] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 03:06:21.811110 7912 tablet_bootstrap.cc:492] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d: Bootstrap starting.
I20260620 03:06:21.816773 7912 tablet_bootstrap.cc:654] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:21.817224 7662 heartbeater.cc:503] Master 127.0.97.60:45945 was elected leader, sending a full tablet report...
I20260620 03:06:21.826500 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.814212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:59203)
0620 03:06:21.825946 (+ 11734us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:21.826146 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11566,"threads_started":1}
I20260620 03:06:21.827711 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.827362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:41957)
0620 03:06:21.827512 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:21.827586 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 03:06:21.827780 7917 tablet_bootstrap.cc:492] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853: Bootstrap starting.
I20260620 03:06:21.829005 7912 tablet_bootstrap.cc:492] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d: No bootstrap required, opened a new log
I20260620 03:06:21.829157 7912 ts_tablet_manager.cc:1397] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.000s
I20260620 03:06:21.829555 7917 tablet_bootstrap.cc:654] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:21.830006 7912 raft_consensus.cc:348] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } } peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260620 03:06:21.830252 7912 raft_consensus.cc:374] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:21.830350 7912 raft_consensus.cc:729] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63870f6d79a144b7bd7696c4c4c47b4d, State: Initialized, Role: FOLLOWER
I20260620 03:06:21.830619 7912 consensus_queue.cc:260] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } } peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260620 03:06:21.831427 7912 ts_tablet_manager.cc:1428] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 03:06:21.831714 7917 tablet_bootstrap.cc:492] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853: No bootstrap required, opened a new log
I20260620 03:06:21.831849 7917 ts_tablet_manager.cc:1397] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 03:06:21.833410 7515 heartbeater.cc:503] Master 127.0.97.60:45945 was elected leader, sending a full tablet report...
I20260620 03:06:21.833496 7917 raft_consensus.cc:348] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } } peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260620 03:06:21.833745 7917 raft_consensus.cc:374] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:21.833853 7917 raft_consensus.cc:729] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4a1d8ef8d9b4cc391b7a657fb81a853, State: Initialized, Role: FOLLOWER
I20260620 03:06:21.834147 7917 consensus_queue.cc:260] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } } peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260620 03:06:21.835855 7917 ts_tablet_manager.cc:1428] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 03:06:21.837517 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.836115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:46793)
0620 03:06:21.836631 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:21.836767 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":551,"threads_started":1}
I20260620 03:06:21.837988 7803 heartbeater.cc:503] Master 127.0.97.60:45945 was elected leader, sending a full tablet report...
I20260620 03:06:21.838415 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.838033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:58529)
0620 03:06:21.838201 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:21.838288 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 03:06:21.840040 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:21.832039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:51725)
0620 03:06:21.839687 (+ 7648us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:21.839816 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7523,"thread_start_us":7429,"threads_started":1}
W20260620 03:06:22.009174 7665 tablet.cc:2369] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:06:22.156627 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:22.156211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:47756)
0620 03:06:22.156393 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:22.156476 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 03:06:22.157640 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:22.157253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:47772)
0620 03:06:22.157395 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:22.157477 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 03:06:22.170485 7919 raft_consensus.cc:482] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:06:22.170679 7919 raft_consensus.cc:504] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } } peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260620 03:06:22.171335 7919 leader_election.cc:288] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b56f067b01d34ef1b8c339beaf862a84 (127.0.97.2:35015), c4a1d8ef8d9b4cc391b7a657fb81a853 (127.0.97.3:35005)
I20260620 03:06:22.171886 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0f5f67f87d4fa2b5702455cfe58a2f" candidate_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b56f067b01d34ef1b8c339beaf862a84" is_pre_election: true
I20260620 03:06:22.171905 7758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0f5f67f87d4fa2b5702455cfe58a2f" candidate_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" is_pre_election: true
I20260620 03:06:22.172144 7617 raft_consensus.cc:2454] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63870f6d79a144b7bd7696c4c4c47b4d in term 0.
I20260620 03:06:22.172142 7758 raft_consensus.cc:2454] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63870f6d79a144b7bd7696c4c4c47b4d in term 0.
I20260620 03:06:22.172634 7404 leader_election.cc:302] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63870f6d79a144b7bd7696c4c4c47b4d, b56f067b01d34ef1b8c339beaf862a84; no voters:
I20260620 03:06:22.172923 7919 raft_consensus.cc:2781] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:06:22.173043 7919 raft_consensus.cc:482] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:06:22.173132 7919 raft_consensus.cc:3037] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:22.175203 7919 raft_consensus.cc:504] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } } peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260620 03:06:22.175889 7919 leader_election.cc:288] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [CANDIDATE]: Term 1 election: Requested vote from peers b56f067b01d34ef1b8c339beaf862a84 (127.0.97.2:35015), c4a1d8ef8d9b4cc391b7a657fb81a853 (127.0.97.3:35005)
I20260620 03:06:22.176362 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0f5f67f87d4fa2b5702455cfe58a2f" candidate_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b56f067b01d34ef1b8c339beaf862a84"
I20260620 03:06:22.176384 7758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0f5f67f87d4fa2b5702455cfe58a2f" candidate_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853"
I20260620 03:06:22.176594 7617 raft_consensus.cc:3037] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:22.176594 7758 raft_consensus.cc:3037] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:22.179530 7617 raft_consensus.cc:2454] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63870f6d79a144b7bd7696c4c4c47b4d in term 1.
I20260620 03:06:22.179530 7758 raft_consensus.cc:2454] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63870f6d79a144b7bd7696c4c4c47b4d in term 1.
I20260620 03:06:22.179983 7404 leader_election.cc:302] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 63870f6d79a144b7bd7696c4c4c47b4d, b56f067b01d34ef1b8c339beaf862a84, c4a1d8ef8d9b4cc391b7a657fb81a853; no voters:
I20260620 03:06:22.180310 7919 raft_consensus.cc:2781] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:22.180800 7919 raft_consensus.cc:686] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [term 1 LEADER]: Becoming Leader. State: Replica: 63870f6d79a144b7bd7696c4c4c47b4d, State: Running, Role: LEADER
I20260620 03:06:22.181111 7919 consensus_queue.cc:237] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } } peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260620 03:06:22.181996 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:22.181420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:43541)
0620 03:06:22.181629 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:22.181817 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 03:06:22.183987 7335 catalog_manager.cc:5495] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 63870f6d79a144b7bd7696c4c4c47b4d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63870f6d79a144b7bd7696c4c4c47b4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } health_report { overall_health: UNKNOWN } } }
I20260620 03:06:22.257915 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:22.252913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34246 (local address 127.0.97.1:38365)
0620 03:06:22.253094 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:22.253098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:22.253115 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:06:22.253249 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:22.253252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:22.253316 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:22.253389 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:06:22.253755 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:22.254427 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:22.256252 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:22.256688 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:22.256784 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:22.257240 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:06:22.257345 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:06:22.257424 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:22.257573 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 03:06:22.257642 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:06:22.260931 7617 raft_consensus.cc:1261] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84 [term 1 FOLLOWER]: Refusing update from remote peer 63870f6d79a144b7bd7696c4c4c47b4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:06:22.260977 7758 raft_consensus.cc:1261] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853 [term 1 FOLLOWER]: Refusing update from remote peer 63870f6d79a144b7bd7696c4c4c47b4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:06:22.261562 7923 consensus_queue.cc:1035] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b56f067b01d34ef1b8c339beaf862a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:22.261955 7919 consensus_queue.cc:1035] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1d8ef8d9b4cc391b7a657fb81a853" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:22.264204 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:22.263737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:48761)
0620 03:06:22.263898 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:22.263989 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 03:06:22.264456 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:22.263776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:37049)
0620 03:06:22.264084 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:22.264223 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260620 03:06:22.270948 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:22.270487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:40103)
0620 03:06:22.270717 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:22.270807 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260620 03:06:22.271500 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:22.270816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:59417)
0620 03:06:22.271156 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:22.271268 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260620 03:06:22.277957 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:22.277557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:47583)
0620 03:06:22.277754 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:22.277829 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 03:06:22.279534 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:22.279042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:60531)
0620 03:06:22.279250 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:22.279382 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 03:06:22.282754 7335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:06:22.282972 7335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:06:22.288036 7335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7ed128efced40ed8d9d0b9fd4ada0f5: Sending DeleteTablet for 3 replicas of tablet 5b0f5f67f87d4fa2b5702455cfe58a2f
I20260620 03:06:22.288897 7450 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b0f5f67f87d4fa2b5702455cfe58a2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:22 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:34242
I20260620 03:06:22.289045 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b0f5f67f87d4fa2b5702455cfe58a2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:22 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:57018
I20260620 03:06:22.289283 7738 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b0f5f67f87d4fa2b5702455cfe58a2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:22 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:33588
I20260620 03:06:22.291110 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 7314
I20260620 03:06:22.301163 7937 tablet_replica.cc:330] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853: stopping tablet replica
I20260620 03:06:22.301419 7937 raft_consensus.cc:2229] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:06:22.301733 7937 raft_consensus.cc:2258] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:22.303004 7935 tablet_replica.cc:330] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d: stopping tablet replica
I20260620 03:06:22.303396 7935 raft_consensus.cc:2229] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:06:22.303543 7937 ts_tablet_manager.cc:1905] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:22.304006 7935 raft_consensus.cc:2258] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:22.304769 7936 tablet_replica.cc:330] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84: stopping tablet replica
I20260620 03:06:22.304950 7936 raft_consensus.cc:2229] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:06:22.305202 7936 raft_consensus.cc:2258] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:22.305648 7935 ts_tablet_manager.cc:1905] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:22.306119 7936 ts_tablet_manager.cc:1905] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:22.310168 7937 ts_tablet_manager.cc:1918] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:06:22.310307 7937 log.cc:1196] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/5b0f5f67f87d4fa2b5702455cfe58a2f
I20260620 03:06:22.310724 7937 ts_tablet_manager.cc:1939] T 5b0f5f67f87d4fa2b5702455cfe58a2f P c4a1d8ef8d9b4cc391b7a657fb81a853: Deleting consensus metadata
I20260620 03:06:22.311822 7323 catalog_manager.cc:4841] TS c4a1d8ef8d9b4cc391b7a657fb81a853 (127.0.97.3:35005): tablet 5b0f5f67f87d4fa2b5702455cfe58a2f (table test-table [id=3622a457b9164cd1a25a0fbc9bfe981e]) successfully deleted
I20260620 03:06:22.312429 7936 ts_tablet_manager.cc:1918] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:06:22.312479 7935 ts_tablet_manager.cc:1918] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:06:22.312582 7936 log.cc:1196] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/5b0f5f67f87d4fa2b5702455cfe58a2f
I20260620 03:06:22.312604 7935 log.cc:1196] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/5b0f5f67f87d4fa2b5702455cfe58a2f
I20260620 03:06:22.313020 7935 ts_tablet_manager.cc:1939] T 5b0f5f67f87d4fa2b5702455cfe58a2f P 63870f6d79a144b7bd7696c4c4c47b4d: Deleting consensus metadata
I20260620 03:06:22.313027 7936 ts_tablet_manager.cc:1939] T 5b0f5f67f87d4fa2b5702455cfe58a2f P b56f067b01d34ef1b8c339beaf862a84: Deleting consensus metadata
I20260620 03:06:22.314014 7323 catalog_manager.cc:4841] TS 63870f6d79a144b7bd7696c4c4c47b4d (127.0.97.1:38365): tablet 5b0f5f67f87d4fa2b5702455cfe58a2f (table test-table [id=3622a457b9164cd1a25a0fbc9bfe981e]) successfully deleted
I20260620 03:06:22.314088 7321 catalog_manager.cc:4841] TS b56f067b01d34ef1b8c339beaf862a84 (127.0.97.2:35015): tablet 5b0f5f67f87d4fa2b5702455cfe58a2f (table test-table [id=3622a457b9164cd1a25a0fbc9bfe981e]) successfully deleted
I20260620 03:06:22.320329 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:22.293523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34250 (local address 127.0.97.1:38365)
0620 03:06:22.293741 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:22.293750 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:22.304589 (+ 10839us) server_negotiation.cc:407] Connection header received
0620 03:06:22.305007 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:22.305014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:22.305099 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:22.305165 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:22.307452 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:22.308560 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:22.310345 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:22.311029 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:22.311160 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:22.314703 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:22.314723 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:22.314746 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:22.314775 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:22.317577 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:22.318270 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:22.318277 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:22.318279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:22.318402 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:22.318773 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:22.318782 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:22.318784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:22.319106 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:22.319460 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:22.319767 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 03:06:22.319975 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 03:06:22.320968 7510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:22.439018 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 7314
I20260620 03:06:22.460803 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 7527
I20260620 03:06:22.473834 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:22.461594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57028 (local address 127.0.97.2:35015)
0620 03:06:22.462096 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:22.462101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:22.462118 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:06:22.462170 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:22.462173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:22.462241 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:22.462336 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:22.463893 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:22.464602 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:22.465888 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:22.466262 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:22.466357 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:22.468797 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:22.468811 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:22.468826 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:22.468869 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:22.471481 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:22.472059 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:22.472066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:22.472068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:22.472204 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:22.472504 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:22.472507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:22.472509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:22.472858 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:22.473057 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:22.473292 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 03:06:22.473442 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":248,"threads_started":1}
I20260620 03:06:22.474619 7657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:22.549299 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:22.548794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:47784)
0620 03:06:22.548996 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:22.549100 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 03:06:22.550365 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:22.549969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:47800)
0620 03:06:22.550132 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:22.550212 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 03:06:22.566500 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 7527
I20260620 03:06:22.583421 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 7675
I20260620 03:06:22.596037 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:22.584187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.0.97.3:35005)
0620 03:06:22.584645 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:22.584651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:22.584669 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:06:22.584722 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:22.584726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:22.584792 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:22.584885 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:22.586135 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:22.586821 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:22.588134 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:22.588517 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:22.588607 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:22.591018 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:22.591032 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:22.591047 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:22.591077 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:22.593808 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:22.594380 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:22.594387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:22.594389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:22.594522 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:22.594837 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:22.594840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:22.594842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:22.595159 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:22.595450 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:22.595621 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 03:06:22.595788 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":265,"threads_started":1}
I20260620 03:06:22.596665 7798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:22.688253 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 7675
I20260620 03:06:22.705644 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 7007
I20260620 03:06:22.706787 7178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:22.820166 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 7007
I20260620 03:06:22.840868 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 7105
I20260620 03:06:22.841874 7366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:22.966475 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:22.966094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44195 (local address 127.0.0.1:58014)
0620 03:06:22.966244 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:22.966331 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 03:06:22.966506 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 7105
2026-06-20T03:06:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11710 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:06:23 dist-test-slave-xv68 krb5kdc[7948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:06:23 dist-test-slave-xv68 krb5kdc[7948](info): set up 2 sockets
Jun 20 03:06:23 dist-test-slave-xv68 krb5kdc[7948](info): commencing operation
krb5kdc: starting...
W20260620 03:06:24.068251 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:06:24 dist-test-slave-xv68 krb5kdc[7948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:06:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:06:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 03:06:24.280828 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36125
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:24.596014 7961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:24.596280 7961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:24.596437 7961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:24.603578 7961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:06:24.603672 7961 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 03:06:24.603736 7961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:24.603835 7961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:24.603916 7961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:06:24.604019 7961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:06:24.613610 7961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36125
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:24.614832 7961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:24.616161 7961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:24.627609 7966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:24.627581 7967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:24.627852 7969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:24.628706 7961 server_base.cc:1034] running on GCE node
I20260620 03:06:24.629444 7961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:24.630738 7961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:24.631978 7961 hybrid_clock.cc:648] HybridClock initialized: now 1781924784631922 us; error 55 us; skew 500 ppm
I20260620 03:06:24.632292 7961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:24 dist-test-slave-xv68 krb5kdc[7948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924784, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:24.635447 7961 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 03:06:24.636729 7961 webserver.cc:460] Webserver started at http://127.0.97.62:39071/ using document root <none> and password file <none>
I20260620 03:06:24.637342 7961 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:24.637423 7961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:24.637681 7961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:24.640465 7961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "e1f760ed552844808735fcfad9de87c7"
format_stamp: "Formatted at 2026-06-20 03:06:24 on dist-test-slave-xv68"
server_key: "a8775ffed68063d419d4c2c5aa843a3c"
server_key_iv: "5b74ef6f38e02bda63e93ed6571624b3"
server_key_version: "encryptionkey@0"
I20260620 03:06:24.641175 7961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "e1f760ed552844808735fcfad9de87c7"
format_stamp: "Formatted at 2026-06-20 03:06:24 on dist-test-slave-xv68"
server_key: "a8775ffed68063d419d4c2c5aa843a3c"
server_key_iv: "5b74ef6f38e02bda63e93ed6571624b3"
server_key_version: "encryptionkey@0"
I20260620 03:06:24.645862 7961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 03:06:24.648993 7976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:24.650221 7961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 03:06:24.650347 7961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "e1f760ed552844808735fcfad9de87c7"
format_stamp: "Formatted at 2026-06-20 03:06:24 on dist-test-slave-xv68"
server_key: "a8775ffed68063d419d4c2c5aa843a3c"
server_key_iv: "5b74ef6f38e02bda63e93ed6571624b3"
server_key_version: "encryptionkey@0"
I20260620 03:06:24.650506 7961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:24.664173 7961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:24.668200 7961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:24.668398 7961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:24.688789 7961 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36125
I20260620 03:06:24.688808 8037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36125 every 8 connection(s)
I20260620 03:06:24.690097 7961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:06:24.693570 8038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:24.700016 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 7961
I20260620 03:06:24.700220 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:06:24.700838 388 external_mini_cluster.cc:1428] Setting key 825d75d4fcaa49fe33fee8ef80ae1016
I20260620 03:06:24.703634 8038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7: Bootstrap starting.
I20260620 03:06:24.707554 8038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:24.708624 8038 log.cc:824] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:24.711440 8038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7: No bootstrap required, opened a new log
Jun 20 03:06:24 dist-test-slave-xv68 krb5kdc[7948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 03:06:24.718798 8038 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f760ed552844808735fcfad9de87c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36125 } }
I20260620 03:06:24.719169 8038 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:24.719336 8038 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f760ed552844808735fcfad9de87c7, State: Initialized, Role: FOLLOWER
I20260620 03:06:24.719888 8038 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f760ed552844808735fcfad9de87c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36125 } }
I20260620 03:06:24.720129 8038 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:06:24.720265 8038 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:06:24.720458 8038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:24.723125 8038 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f760ed552844808735fcfad9de87c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36125 } }
I20260620 03:06:24.723423 8041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:24.702793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52084 (local address 127.0.97.62:36125)
0620 03:06:24.703596 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:24.703612 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:24.703668 (+ 56us) server_negotiation.cc:407] Connection header received
0620 03:06:24.706108 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:24.706135 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:24.706581 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:24.706986 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:24.708438 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:24.710273 (+ 1835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:24.711606 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:24.712272 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:24.712465 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:24.716195 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:24.716202 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:24.716216 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:24.716252 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:24.719343 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:24.720097 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:24.720118 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:24.720125 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:24.720338 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:24.720790 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:24.720794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:24.720796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:24.721339 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:24.721588 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:24.722010 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 03:06:24.722393 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":355,"threads_started":1}
I20260620 03:06:24.723719 8038 leader_election.cc:302] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1f760ed552844808735fcfad9de87c7; no voters:
I20260620 03:06:24.724371 8038 leader_election.cc:288] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:06:24.724476 8043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:24.725564 8043 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7 [term 1 LEADER]: Becoming Leader. State: Replica: e1f760ed552844808735fcfad9de87c7, State: Running, Role: LEADER
I20260620 03:06:24.726101 8043 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f760ed552844808735fcfad9de87c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36125 } }
I20260620 03:06:24.726650 8038 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:06:24.729071 8045 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1f760ed552844808735fcfad9de87c7. Latest consensus state: current_term: 1 leader_uuid: "e1f760ed552844808735fcfad9de87c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f760ed552844808735fcfad9de87c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36125 } } }
I20260620 03:06:24.729313 8044 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1f760ed552844808735fcfad9de87c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f760ed552844808735fcfad9de87c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36125 } } }
I20260620 03:06:24.729815 8045 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:24.729832 8044 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7 [sys.catalog]: This master's current role is: LEADER
W20260620 03:06:24.741868 8059 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 03:06:24.741988 8059 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 03:06:24.742125 8051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:06:24.744256 8051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:06:24.752751 8051 catalog_manager.cc:1347] Generated new cluster ID: 1e09546cf8844a3391994027e221ef6c
I20260620 03:06:24.752874 8051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:06:24.780683 8051 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:06:24.782486 8051 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:06:24.793083 8051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7: Generated new TSK 0
I20260620 03:06:24.794059 8051 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 03:06:24.853067 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36125
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:25.197870 8064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:25.198140 8064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:25.198247 8064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:25.205580 8064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:25.205695 8064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:25.205782 8064 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 03:06:25.205854 8064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:06:25.215754 8064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:25.216956 8064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:25.218215 8064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:25.231710 8069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:25.231889 8072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:25.231716 8070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:25.232731 8064 server_base.cc:1034] running on GCE node
I20260620 03:06:25.233384 8064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:25.234333 8064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:25.235666 8064 hybrid_clock.cc:648] HybridClock initialized: now 1781924785235577 us; error 48 us; skew 500 ppm
I20260620 03:06:25.236212 8064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:25 dist-test-slave-xv68 krb5kdc[7948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924785, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:25.239655 8064 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 03:06:25.241065 8064 webserver.cc:460] Webserver started at http://127.0.97.1:41475/ using document root <none> and password file <none>
I20260620 03:06:25.241833 8064 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:25.241967 8064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:25.242287 8064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:25.245225 8064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "92b0699d42404289801130a1359f1270"
format_stamp: "Formatted at 2026-06-20 03:06:25 on dist-test-slave-xv68"
server_key: "d9a7b9ea17eedb4efc9364f7b65cab07"
server_key_iv: "752b479172a7631aa90245140de6d8a3"
server_key_version: "encryptionkey@0"
I20260620 03:06:25.246034 8064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "92b0699d42404289801130a1359f1270"
format_stamp: "Formatted at 2026-06-20 03:06:25 on dist-test-slave-xv68"
server_key: "d9a7b9ea17eedb4efc9364f7b65cab07"
server_key_iv: "752b479172a7631aa90245140de6d8a3"
server_key_version: "encryptionkey@0"
I20260620 03:06:25.251257 8064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 03:06:25.255005 8079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:25.256489 8064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:06:25.256683 8064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "92b0699d42404289801130a1359f1270"
format_stamp: "Formatted at 2026-06-20 03:06:25 on dist-test-slave-xv68"
server_key: "d9a7b9ea17eedb4efc9364f7b65cab07"
server_key_iv: "752b479172a7631aa90245140de6d8a3"
server_key_version: "encryptionkey@0"
I20260620 03:06:25.256942 8064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:25.272085 8064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:25.276562 8064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:25.276919 8064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:25.279838 8064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:25.279945 8064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:25.280081 8064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:25.280177 8064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 03:06:25 dist-test-slave-xv68 krb5kdc[7948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924785, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 03:06:25.304196 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:25.280470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42107 (local address 127.0.97.62:36125)
0620 03:06:25.280978 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:25.280986 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:25.282393 (+ 1407us) server_negotiation.cc:407] Connection header received
0620 03:06:25.284557 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:25.284563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:25.284654 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:25.284777 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:25.287159 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:25.288197 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:25.291669 (+ 3472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:25.292375 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:25.292528 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:25.296487 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:25.296493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:25.296505 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:25.296535 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:25.299888 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:25.300612 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:25.300618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:25.300629 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:25.300747 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:25.301314 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:25.301319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:25.301322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:25.301582 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:25.301773 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:25.303493 (+ 1720us) server_negotiation.cc:299] Negotiation successful
0620 03:06:25.303758 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":299,"threads_started":1}
I20260620 03:06:25.305503 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:25.281428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36125 (local address 127.0.97.1:42107)
0620 03:06:25.282211 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:25.282254 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:25.284283 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:25.284999 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:25.285007 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:25.285732 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:25.286969 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:25.286993 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:25.288417 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:25.288426 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:25.291482 (+ 3056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:25.291503 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:25.292661 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:25.292670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:25.292951 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:25.293679 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:25.293710 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:25.296259 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:25.300021 (+ 3762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:25.300032 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:25.300051 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:25.300482 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:25.300856 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:25.300867 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:25.300870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:25.301200 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:25.301777 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:25.301788 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:25.302360 (+ 572us) client_negotiation.cc:769] Sending connection context
0620 03:06:25.302814 (+ 454us) client_negotiation.cc:240] Negotiation successful
0620 03:06:25.303536 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":358,"threads_started":1}
I20260620 03:06:25.330070 8064 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46135
I20260620 03:06:25.330093 8199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46135 every 8 connection(s)
I20260620 03:06:25.331477 8064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:06:25.332621 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:25.311764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43915 (local address 127.0.97.62:36125)
0620 03:06:25.311966 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:25.311974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:25.319967 (+ 7993us) server_negotiation.cc:407] Connection header received
0620 03:06:25.320249 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:25.320257 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:25.320327 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:25.320432 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:25.321344 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:25.322058 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:25.323411 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:25.323933 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:25.324077 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:25.326571 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:25.326577 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:25.326580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:25.326609 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:25.329835 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:25.330624 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:25.330631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:25.330634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:25.330763 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:25.331117 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:25.331124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:25.331127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:25.331461 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:25.331607 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:25.332027 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 03:06:25.332248 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 03:06:25.332770 8162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:25.311836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36125 (local address 127.0.97.1:43915)
0620 03:06:25.319821 (+ 7985us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:25.319847 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:25.320035 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:25.320459 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:25.320464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:25.320915 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:25.321198 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:25.321207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:25.322200 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:25.322206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:25.323216 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:25.323226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:25.324171 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:25.324177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:25.324447 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:25.325246 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:25.325278 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:25.326389 (+ 1111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:25.329974 (+ 3585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:25.329979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:25.329986 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:25.330506 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:25.330872 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:25.330878 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:25.330881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:25.331005 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:25.331583 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:25.331588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:25.331777 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 03:06:25.331914 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 03:06:25.332260 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7830,"thread_start_us":7689,"threads_started":1}
I20260620 03:06:25.334157 8200 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36125
I20260620 03:06:25.334431 8200 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:25.334798 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 8064
I20260620 03:06:25.334992 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:06:25.335314 8200 heartbeater.cc:511] Master 127.0.97.62:36125 requested a full tablet report, sending...
I20260620 03:06:25.335834 388 external_mini_cluster.cc:1428] Setting key f38d93c03dc4f164d6b94edd9c76812d
I20260620 03:06:25.337921 7993 ts_manager.cc:194] Registered new tserver with Master: 92b0699d42404289801130a1359f1270 (127.0.97.1:46135)
I20260620 03:06:25.339675 7993 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42107
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 03:06:25.380604 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36125
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:25.707746 8204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:25.708101 8204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:25.708319 8204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:25.716977 8204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:25.717082 8204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:25.717172 8204 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 03:06:25.717252 8204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:06:25.726881 8204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:25.728297 8204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:25.729620 8204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:25.742439 8210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:25.742595 8209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:25.742647 8212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:25.743544 8204 server_base.cc:1034] running on GCE node
I20260620 03:06:25.744110 8204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:25.744931 8204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:25.746140 8204 hybrid_clock.cc:648] HybridClock initialized: now 1781924785746082 us; error 30 us; skew 500 ppm
I20260620 03:06:25.746429 8204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:25 dist-test-slave-xv68 krb5kdc[7948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924785, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:25.749477 8204 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 03:06:25.750774 8204 webserver.cc:460] Webserver started at http://127.0.97.2:46799/ using document root <none> and password file <none>
I20260620 03:06:25.751494 8204 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:25.751619 8204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:25.751925 8204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:25.754834 8204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "b4b81c6b987b42018936bd56b0862748"
format_stamp: "Formatted at 2026-06-20 03:06:25 on dist-test-slave-xv68"
server_key: "ff9207cc33a752d91f72f0d69c509d72"
server_key_iv: "6d1e8a397db9a8831bc81e873e687329"
server_key_version: "encryptionkey@0"
I20260620 03:06:25.755658 8204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b4b81c6b987b42018936bd56b0862748"
format_stamp: "Formatted at 2026-06-20 03:06:25 on dist-test-slave-xv68"
server_key: "ff9207cc33a752d91f72f0d69c509d72"
server_key_iv: "6d1e8a397db9a8831bc81e873e687329"
server_key_version: "encryptionkey@0"
I20260620 03:06:25.760578 8204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:06:25.763931 8219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:25.765168 8204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:06:25.765349 8204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "b4b81c6b987b42018936bd56b0862748"
format_stamp: "Formatted at 2026-06-20 03:06:25 on dist-test-slave-xv68"
server_key: "ff9207cc33a752d91f72f0d69c509d72"
server_key_iv: "6d1e8a397db9a8831bc81e873e687329"
server_key_version: "encryptionkey@0"
I20260620 03:06:25.765578 8204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:25.782109 8204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:25.785885 8204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:25.786162 8204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:25.788730 8204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:25.788841 8204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:25.788965 8204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:25.789021 8204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 03:06:25 dist-test-slave-xv68 krb5kdc[7948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924785, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 03:06:25.813959 8228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:25.789691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36125 (local address 127.0.97.2:35771)
0620 03:06:25.790552 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:25.790598 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:25.792701 (+ 2103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:25.793424 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:25.793435 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:25.794406 (+ 971us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:25.795428 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:25.795467 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:25.796795 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:25.796807 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:25.799652 (+ 2845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:25.799676 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:25.800749 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:25.800759 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:25.801132 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:25.801997 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:25.802018 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:25.805234 (+ 3216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:25.809347 (+ 4113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:25.809358 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:25.809381 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:25.810007 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:25.810379 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:25.810384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:25.810388 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:25.810577 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:25.811112 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:25.811119 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:25.811504 (+ 385us) client_negotiation.cc:769] Sending connection context
0620 03:06:25.811816 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 03:06:25.812410 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":381,"threads_started":1}
I20260620 03:06:25.815320 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:25.789037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35771 (local address 127.0.97.62:36125)
0620 03:06:25.789243 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:25.789252 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:25.790734 (+ 1482us) server_negotiation.cc:407] Connection header received
0620 03:06:25.792983 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:25.792989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:25.793081 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:25.793207 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:25.795678 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:25.796623 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:25.799865 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:25.800457 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:25.800593 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:25.805496 (+ 4903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:25.805505 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:25.805513 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:25.805559 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:25.809178 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:25.810137 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:25.810142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:25.810146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:25.810270 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:25.810695 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:25.810700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:25.810704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:25.810990 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:25.811139 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:25.814743 (+ 3604us) server_negotiation.cc:299] Negotiation successful
0620 03:06:25.814946 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 03:06:25.834981 8204 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37803
I20260620 03:06:25.835209 8338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37803 every 8 connection(s)
I20260620 03:06:25.835443 8323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:25.822126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36125 (local address 127.0.97.2:43703)
0620 03:06:25.822801 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:25.822822 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:25.822959 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:25.823265 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:25.823269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:25.823914 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:25.824353 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:25.824365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:25.825322 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:25.825330 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:25.826442 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:25.826453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:25.827185 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:25.827191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:25.827392 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:25.828058 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:25.828075 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:25.828737 (+ 662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:25.832064 (+ 3327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:25.832069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:25.832076 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:25.832692 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:25.833113 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:25.833120 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:25.833126 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:25.833284 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:25.833842 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:25.833847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:25.834078 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 03:06:25.834259 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 03:06:25.834693 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":382,"threads_started":1}
I20260620 03:06:25.835633 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:25.822018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43703 (local address 127.0.97.62:36125)
0620 03:06:25.822190 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:25.822199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:25.822902 (+ 703us) server_negotiation.cc:407] Connection header received
0620 03:06:25.823080 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:25.823086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:25.823151 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:25.823247 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:25.824474 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:25.825186 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:25.826575 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:25.827010 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:25.827146 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:25.828918 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:25.828923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:25.828927 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:25.828954 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:25.831889 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:25.832832 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:25.832841 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:25.832845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:25.832959 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:25.833392 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:25.833398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:25.833402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:25.833712 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:25.833867 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:25.835067 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0620 03:06:25.835312 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:06:25.836648 8204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:06:25.839367 8339 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36125
I20260620 03:06:25.839622 8339 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:25.840272 8339 heartbeater.cc:511] Master 127.0.97.62:36125 requested a full tablet report, sending...
I20260620 03:06:25.840492 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 8204
I20260620 03:06:25.840647 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:06:25.841080 388 external_mini_cluster.cc:1428] Setting key d5b82de6198d78f33558dafcb67ab758
I20260620 03:06:25.841634 7993 ts_manager.cc:194] Registered new tserver with Master: b4b81c6b987b42018936bd56b0862748 (127.0.97.2:37803)
I20260620 03:06:25.842764 7993 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35771
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 03:06:25.883877 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36125
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:26.205260 8343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:26.205569 8343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:26.205746 8343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:26.213007 8343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:26.213148 8343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:26.213301 8343 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 03:06:26.213438 8343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:06:26.222891 8343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:26.224184 8343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:26.225549 8343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:26.238538 8349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:26.238605 8348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:26.238574 8351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:26.240475 8343 server_base.cc:1034] running on GCE node
I20260620 03:06:26.241094 8343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:26.242023 8343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:26.243340 8343 hybrid_clock.cc:648] HybridClock initialized: now 1781924786243210 us; error 56 us; skew 500 ppm
I20260620 03:06:26.243652 8343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:26 dist-test-slave-xv68 krb5kdc[7948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924786, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:26.246840 8343 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 03:06:26.248206 8343 webserver.cc:460] Webserver started at http://127.0.97.3:41057/ using document root <none> and password file <none>
I20260620 03:06:26.248919 8343 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:26.249047 8343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:26.249362 8343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:26.252400 8343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "d3ebc275dcfa4053ad7231d7967a6c4f"
format_stamp: "Formatted at 2026-06-20 03:06:26 on dist-test-slave-xv68"
server_key: "464df4b120831d76524c7e218998fc01"
server_key_iv: "f1289f0265594546d1ab7dc822b0d119"
server_key_version: "encryptionkey@0"
I20260620 03:06:26.253208 8343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d3ebc275dcfa4053ad7231d7967a6c4f"
format_stamp: "Formatted at 2026-06-20 03:06:26 on dist-test-slave-xv68"
server_key: "464df4b120831d76524c7e218998fc01"
server_key_iv: "f1289f0265594546d1ab7dc822b0d119"
server_key_version: "encryptionkey@0"
I20260620 03:06:26.258268 8343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:06:26.261818 8358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:26.263091 8343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 03:06:26.263275 8343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "d3ebc275dcfa4053ad7231d7967a6c4f"
format_stamp: "Formatted at 2026-06-20 03:06:26 on dist-test-slave-xv68"
server_key: "464df4b120831d76524c7e218998fc01"
server_key_iv: "f1289f0265594546d1ab7dc822b0d119"
server_key_version: "encryptionkey@0"
I20260620 03:06:26.263540 8343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:26.279011 8343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:26.283236 8343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:26.283529 8343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:26.286137 8343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:26.286239 8343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:26.286387 8343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:26.286514 8343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 20 03:06:26 dist-test-slave-xv68 krb5kdc[7948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924786, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 03:06:26.311491 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.286552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45477 (local address 127.0.97.62:36125)
0620 03:06:26.286788 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:26.286798 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:26.288258 (+ 1460us) server_negotiation.cc:407] Connection header received
0620 03:06:26.290423 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:26.290428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:26.290525 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:26.290659 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:26.293314 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.294387 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.297611 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.298212 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.298396 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.303411 (+ 5015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:26.303417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:26.303420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:26.303449 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:26.307431 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:26.308486 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:26.308490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:26.308492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:26.308735 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:26.309327 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:26.309330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:26.309332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:26.309598 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:26.309744 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:26.310781 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0620 03:06:26.311034 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 03:06:26.313261 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.287211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36125 (local address 127.0.97.3:45477)
0620 03:06:26.288063 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:26.288104 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:26.290128 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:26.290952 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:26.290966 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:26.291894 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:26.293054 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.293088 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.294592 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.294601 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.297401 (+ 2800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.297423 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.298508 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.298518 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.298915 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.299922 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:26.299966 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:26.303169 (+ 3203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:26.307613 (+ 4444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:26.307624 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:26.307638 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:26.308342 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:26.308883 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:26.308891 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:26.308897 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:26.309208 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:26.309760 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:26.309771 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:26.310274 (+ 503us) client_negotiation.cc:769] Sending connection context
0620 03:06:26.310632 (+ 358us) client_negotiation.cc:240] Negotiation successful
0620 03:06:26.311245 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":399,"threads_started":1}
I20260620 03:06:26.332510 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.319642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55829 (local address 127.0.97.62:36125)
0620 03:06:26.319847 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:26.319854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:26.320498 (+ 644us) server_negotiation.cc:407] Connection header received
0620 03:06:26.320675 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:26.320681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:26.320763 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:26.320860 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:26.321704 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.322376 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.323897 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.324440 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.324597 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.326265 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:26.326271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:26.326276 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:26.326313 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:26.329675 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:26.330634 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:26.330637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:26.330639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:26.330749 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:26.331097 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:26.331105 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:26.331108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:26.331428 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:26.331565 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:26.331970 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 03:06:26.332177 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 03:06:26.332643 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.319725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36125 (local address 127.0.97.3:55829)
0620 03:06:26.320404 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:26.320419 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:26.320560 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:26.320880 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:26.320884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:26.321299 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:26.321560 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.321568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.322508 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.322514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.323561 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.323575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.324632 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.324638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.324815 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.325411 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:26.325427 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:26.326078 (+ 651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:26.329856 (+ 3778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:26.329863 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:26.329870 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:26.330452 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:26.330857 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:26.330862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:26.330865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:26.330990 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:26.331544 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:26.331549 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:26.331726 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 03:06:26.331865 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 03:06:26.332175 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":391,"threads_started":1}
I20260620 03:06:26.342263 8200 heartbeater.cc:503] Master 127.0.97.62:36125 was elected leader, sending a full tablet report...
I20260620 03:06:26.351699 8343 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33765
I20260620 03:06:26.351718 8477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33765 every 8 connection(s)
I20260620 03:06:26.352910 8343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:06:26.354861 8478 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36125
I20260620 03:06:26.355041 8478 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:26.355706 8478 heartbeater.cc:511] Master 127.0.97.62:36125 requested a full tablet report, sending...
I20260620 03:06:26.355953 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 8343
I20260620 03:06:26.356105 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:06:26.356623 388 external_mini_cluster.cc:1428] Setting key 6c67de9b0aa9375c7866540ba3b2d62b
I20260620 03:06:26.357023 7993 ts_manager.cc:194] Registered new tserver with Master: d3ebc275dcfa4053ad7231d7967a6c4f (127.0.97.3:33765)
I20260620 03:06:26.358052 7993 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45477
I20260620 03:06:26.359560 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 03:06:26 dist-test-slave-xv68 krb5kdc[7948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924786, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 03:06:26 dist-test-slave-xv68 krb5kdc[7948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:26 dist-test-slave-xv68 krb5kdc[7948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 03:06:26.413669 8088 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:26.397741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52094 (local address 127.0.97.62:36125)
0620 03:06:26.397936 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:26.397943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:26.398111 (+ 168us) server_negotiation.cc:407] Connection header received
0620 03:06:26.398495 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:26.398501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:26.398604 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:26.398766 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:26.400800 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.402137 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.404268 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.404834 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.404960 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.413400 (+ 8440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52094: BlockingRecv error: recv got EOF from 127.0.0.1:52094 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 03:06:26 dist-test-slave-xv68 krb5kdc[7948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924786, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 03:06:26.432638 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.417357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.0.97.62:36125)
0620 03:06:26.417531 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:26.417536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:26.417855 (+ 319us) server_negotiation.cc:407] Connection header received
0620 03:06:26.418071 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:26.418076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:26.418160 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:26.418243 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:26.419855 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.420538 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.422533 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.423274 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.423431 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.427472 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:26.427478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:26.427483 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:26.427524 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:26.430123 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:26.430771 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:26.430778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:26.430782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:26.430907 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:26.431275 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:26.431282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:26.431285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:26.431575 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:26.431864 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:26.432086 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 03:06:26.432299 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 03:06:26.436303 7993 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:06:26.443069 7993 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:06:26.487517 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.454182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37803 (local address 127.0.0.1:44402)
0620 03:06:26.479709 (+ 25527us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:26.479730 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:26.479922 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:26.480794 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:26.480800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:26.480826 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:26.481349 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.481361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.483105 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.483110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.484789 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.484800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.486405 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.486411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.486598 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.486650 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 03:06:26.486796 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 03:06:26.486978 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25411,"mutex_wait_us":57,"thread_start_us":24814,"threads_started":1}
I20260620 03:06:26.488020 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.453918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33765 (local address 127.0.0.1:35428)
0620 03:06:26.478691 (+ 24773us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:26.478715 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:26.478879 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:26.480069 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:26.480076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:26.480106 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:26.480482 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.480498 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.482592 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.482599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.485099 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.485120 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.486864 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.486870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.487045 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.487084 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 03:06:26.487192 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 03:06:26.487337 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24612,"mutex_wait_us":1,"thread_start_us":25544,"threads_started":1}
I20260620 03:06:26.488059 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.453615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46135 (local address 127.0.0.1:60566)
0620 03:06:26.472802 (+ 19187us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:26.472868 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:26.473097 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:26.480520 (+ 7423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:26.480524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:26.480551 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:26.481079 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.481097 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.482906 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.482910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.485243 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.485254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.487237 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.487243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.487429 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.487474 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 03:06:26.487564 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 03:06:26.487696 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18802,"mutex_wait_us":35,"thread_start_us":19720,"threads_started":1}
I20260620 03:06:26.488258 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.454742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44402 (local address 127.0.97.2:37803)
0620 03:06:26.477883 (+ 23141us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:26.477896 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:26.479824 (+ 1928us) server_negotiation.cc:407] Connection header received
0620 03:06:26.480130 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:26.480140 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:26.480380 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:26.480579 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:26.481547 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.482963 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.484941 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.486218 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.486351 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.486553 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:26.487541 (+ 988us) server_negotiation.cc:299] Negotiation successful
0620 03:06:26.487834 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22931,"thread_start_us":23156,"threads_started":1}
I20260620 03:06:26.488422 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.454620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35428 (local address 127.0.97.3:33765)
0620 03:06:26.463957 (+ 9337us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:26.463970 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:26.478861 (+ 14891us) server_negotiation.cc:407] Connection header received
0620 03:06:26.479094 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:26.479106 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:26.479482 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:26.479709 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:26.480905 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.482464 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.485267 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.486652 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.486789 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.487012 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:26.487247 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 03:06:26.487450 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9092,"thread_start_us":8893,"threads_started":1}
I20260620 03:06:26.488827 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.453795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.0.97.1:46135)
0620 03:06:26.480045 (+ 26250us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:26.480054 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:26.480078 (+ 24us) server_negotiation.cc:407] Connection header received
0620 03:06:26.480167 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:26.480171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:26.480340 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:26.480527 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:26.481282 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.482773 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.485401 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.486802 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.486913 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.487093 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:26.488361 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0620 03:06:26.488528 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26087,"thread_start_us":25954,"threads_started":1}
I20260620 03:06:26.491917 8130 tablet_service.cc:1467] Processing CreateTablet for tablet 31f6298f4161462ebb3d8cd7df970761 (DEFAULT_TABLE table=test-table [id=79d599e342d24bf5bf5076a9020ec386]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:26.491914 8408 tablet_service.cc:1467] Processing CreateTablet for tablet 31f6298f4161462ebb3d8cd7df970761 (DEFAULT_TABLE table=test-table [id=79d599e342d24bf5bf5076a9020ec386]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:26.491914 8269 tablet_service.cc:1467] Processing CreateTablet for tablet 31f6298f4161462ebb3d8cd7df970761 (DEFAULT_TABLE table=test-table [id=79d599e342d24bf5bf5076a9020ec386]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:26.493156 8130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f6298f4161462ebb3d8cd7df970761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:26.493228 8269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f6298f4161462ebb3d8cd7df970761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:26.493234 8408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f6298f4161462ebb3d8cd7df970761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:26.503629 8502 tablet_bootstrap.cc:492] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270: Bootstrap starting.
I20260620 03:06:26.507010 8502 tablet_bootstrap.cc:654] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:26.508231 8502 log.cc:824] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:26.510010 8504 tablet_bootstrap.cc:492] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f: Bootstrap starting.
I20260620 03:06:26.515096 8504 tablet_bootstrap.cc:654] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:26.516063 8503 tablet_bootstrap.cc:492] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748: Bootstrap starting.
I20260620 03:06:26.520157 8503 tablet_bootstrap.cc:654] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:26.520232 8504 log.cc:824] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:26.521387 8503 log.cc:824] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:26.522545 8502 tablet_bootstrap.cc:492] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270: No bootstrap required, opened a new log
I20260620 03:06:26.522842 8502 ts_tablet_manager.cc:1397] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.015s
I20260620 03:06:26.523799 8503 tablet_bootstrap.cc:492] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748: No bootstrap required, opened a new log
I20260620 03:06:26.524035 8503 ts_tablet_manager.cc:1397] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 03:06:26.524291 8504 tablet_bootstrap.cc:492] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f: No bootstrap required, opened a new log
I20260620 03:06:26.524513 8504 ts_tablet_manager.cc:1397] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260620 03:06:26.530856 8503 raft_consensus.cc:348] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } }
I20260620 03:06:26.531206 8503 raft_consensus.cc:374] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:26.531338 8503 raft_consensus.cc:729] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4b81c6b987b42018936bd56b0862748, State: Initialized, Role: FOLLOWER
I20260620 03:06:26.531929 8503 consensus_queue.cc:260] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } }
I20260620 03:06:26.532899 8339 heartbeater.cc:503] Master 127.0.97.62:36125 was elected leader, sending a full tablet report...
I20260620 03:06:26.533324 8502 raft_consensus.cc:348] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } }
I20260620 03:06:26.533722 8502 raft_consensus.cc:374] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:26.533818 8502 raft_consensus.cc:729] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b0699d42404289801130a1359f1270, State: Initialized, Role: FOLLOWER
I20260620 03:06:26.534363 8502 consensus_queue.cc:260] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } }
I20260620 03:06:26.534747 8503 ts_tablet_manager.cc:1428] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260620 03:06:26.534621 8504 raft_consensus.cc:348] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } }
I20260620 03:06:26.535066 8504 raft_consensus.cc:374] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:26.535187 8504 raft_consensus.cc:729] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ebc275dcfa4053ad7231d7967a6c4f, State: Initialized, Role: FOLLOWER
I20260620 03:06:26.535892 8504 consensus_queue.cc:260] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } }
I20260620 03:06:26.537485 8478 heartbeater.cc:503] Master 127.0.97.62:36125 was elected leader, sending a full tablet report...
I20260620 03:06:26.537564 8504 ts_tablet_manager.cc:1428] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 03:06:26.539542 8502 ts_tablet_manager.cc:1428] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270: Time spent starting tablet: real 0.017s user 0.013s sys 0.001s
W20260620 03:06:26.588054 8201 tablet.cc:2369] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:06:26.588846 8340 tablet.cc:2369] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:06:26.604493 8479 tablet.cc:2369] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:06:26.678689 8508 raft_consensus.cc:482] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:06:26.679054 8508 raft_consensus.cc:504] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } }
I20260620 03:06:26.680707 8508 leader_election.cc:288] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92b0699d42404289801130a1359f1270 (127.0.97.1:46135), d3ebc275dcfa4053ad7231d7967a6c4f (127.0.97.3:33765)
I20260620 03:06:26.687201 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.681085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50763 (local address 127.0.97.3:33765)
0620 03:06:26.681259 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:26.681267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:26.682015 (+ 748us) server_negotiation.cc:407] Connection header received
0620 03:06:26.682159 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:26.682185 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:26.682273 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:26.682372 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:26.682995 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.683784 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.685490 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.686205 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.686360 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.686469 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:26.686814 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 03:06:26.686940 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 03:06:26.687233 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.680992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55663 (local address 127.0.97.1:46135)
0620 03:06:26.681246 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:26.681253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:26.681831 (+ 578us) server_negotiation.cc:407] Connection header received
0620 03:06:26.682014 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:26.682022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:26.682113 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:26.682201 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:26.682855 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.683658 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.685433 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.686197 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.686342 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.686457 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:26.686884 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 03:06:26.687010 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260620 03:06:26.687376 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.680913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46135 (local address 127.0.97.2:55663)
0620 03:06:26.681687 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:26.681710 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:26.681874 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:26.682228 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:26.682232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:26.682254 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:26.682699 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.682711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.683796 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.683801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.685276 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.685287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.686429 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.686436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.686684 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.686706 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 03:06:26.686845 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 03:06:26.686960 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":59,"thread_start_us":389,"threads_started":1}
I20260620 03:06:26.687498 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.681085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33765 (local address 127.0.97.2:50763)
0620 03:06:26.681877 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:26.681899 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:26.682039 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:26.682401 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:26.682404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:26.682423 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:26.682838 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.682848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.683917 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.683920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.685323 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.685334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.686465 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.686473 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.686620 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.686687 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 03:06:26.686758 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 03:06:26.686886 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":62,"thread_start_us":465,"threads_started":1}
I20260620 03:06:26.688510 8150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f6298f4161462ebb3d8cd7df970761" candidate_uuid: "b4b81c6b987b42018936bd56b0862748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b0699d42404289801130a1359f1270" is_pre_election: true
I20260620 03:06:26.688510 8428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f6298f4161462ebb3d8cd7df970761" candidate_uuid: "b4b81c6b987b42018936bd56b0862748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" is_pre_election: true
I20260620 03:06:26.689092 8428 raft_consensus.cc:2454] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4b81c6b987b42018936bd56b0862748 in term 0.
I20260620 03:06:26.689092 8150 raft_consensus.cc:2454] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4b81c6b987b42018936bd56b0862748 in term 0.
I20260620 03:06:26.689769 8221 leader_election.cc:302] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92b0699d42404289801130a1359f1270, b4b81c6b987b42018936bd56b0862748; no voters:
I20260620 03:06:26.690222 8508 raft_consensus.cc:2781] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:06:26.690368 8508 raft_consensus.cc:482] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:06:26.690469 8508 raft_consensus.cc:3037] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:26.692696 8508 raft_consensus.cc:504] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } }
I20260620 03:06:26.693310 8508 leader_election.cc:288] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [CANDIDATE]: Term 1 election: Requested vote from peers 92b0699d42404289801130a1359f1270 (127.0.97.1:46135), d3ebc275dcfa4053ad7231d7967a6c4f (127.0.97.3:33765)
I20260620 03:06:26.693739 8150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f6298f4161462ebb3d8cd7df970761" candidate_uuid: "b4b81c6b987b42018936bd56b0862748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b0699d42404289801130a1359f1270"
I20260620 03:06:26.693791 8428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f6298f4161462ebb3d8cd7df970761" candidate_uuid: "b4b81c6b987b42018936bd56b0862748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f"
I20260620 03:06:26.693948 8150 raft_consensus.cc:3037] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:26.693957 8428 raft_consensus.cc:3037] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:26.697475 8428 raft_consensus.cc:2454] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4b81c6b987b42018936bd56b0862748 in term 1.
I20260620 03:06:26.697463 8150 raft_consensus.cc:2454] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4b81c6b987b42018936bd56b0862748 in term 1.
I20260620 03:06:26.697911 8222 leader_election.cc:302] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4b81c6b987b42018936bd56b0862748, d3ebc275dcfa4053ad7231d7967a6c4f; no voters:
I20260620 03:06:26.698256 8508 raft_consensus.cc:2781] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:26.698711 8508 raft_consensus.cc:686] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [term 1 LEADER]: Becoming Leader. State: Replica: b4b81c6b987b42018936bd56b0862748, State: Running, Role: LEADER
I20260620 03:06:26.699527 8508 consensus_queue.cc:237] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } }
I20260620 03:06:26.704344 7992 catalog_manager.cc:5495] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4b81c6b987b42018936bd56b0862748 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b4b81c6b987b42018936bd56b0862748" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } health_report { overall_health: HEALTHY } } }
I20260620 03:06:26.721828 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.716754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44412 (local address 127.0.97.2:37803)
0620 03:06:26.716933 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:26.716939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:26.717032 (+ 93us) server_negotiation.cc:407] Connection header received
0620 03:06:26.717255 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:26.717264 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:26.717334 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:26.717488 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:06:26.717858 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.718548 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.719928 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.720325 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.720450 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.720828 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:06:26.721295 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:06:26.721448 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:26.721500 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 03:06:26.721572 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 03:06:26.729218 8428 raft_consensus.cc:1261] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 FOLLOWER]: Refusing update from remote peer b4b81c6b987b42018936bd56b0862748: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:06:26.729239 8150 raft_consensus.cc:1261] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Refusing update from remote peer b4b81c6b987b42018936bd56b0862748: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:06:26.730132 8513 consensus_queue.cc:1035] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:26.730451 8508 consensus_queue.cc:1035] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:26.755750 7992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:06:26.756000 7992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:06:26.759111 7992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7: Sending DeleteTablet for 3 replicas of tablet 31f6298f4161462ebb3d8cd7df970761
I20260620 03:06:26.760145 8130 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f6298f4161462ebb3d8cd7df970761 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:26 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60566
I20260620 03:06:26.760257 8269 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f6298f4161462ebb3d8cd7df970761 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:26 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44402
I20260620 03:06:26.760289 8408 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f6298f4161462ebb3d8cd7df970761 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:26 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35428
I20260620 03:06:26.763762 7993 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:06:26.764540 7993 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:06:26.773651 8129 tablet_service.cc:1467] Processing CreateTablet for tablet 404c6e9ab8424368984521aaec9b08df (DEFAULT_TABLE table=test-table [id=5b31f1fbdbad49c5b06b5c868dca64b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:26.773943 8407 tablet_service.cc:1467] Processing CreateTablet for tablet 404c6e9ab8424368984521aaec9b08df (DEFAULT_TABLE table=test-table [id=5b31f1fbdbad49c5b06b5c868dca64b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:26.774278 8129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404c6e9ab8424368984521aaec9b08df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:26.774355 8528 tablet_replica.cc:330] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f: stopping tablet replica
I20260620 03:06:26.774528 8407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404c6e9ab8424368984521aaec9b08df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:26.774804 8528 raft_consensus.cc:2229] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:06:26.775327 8528 raft_consensus.cc:2258] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:26.779417 8528 ts_tablet_manager.cc:1905] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:26.780030 8268 tablet_service.cc:1467] Processing CreateTablet for tablet 404c6e9ab8424368984521aaec9b08df (DEFAULT_TABLE table=test-table [id=5b31f1fbdbad49c5b06b5c868dca64b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:26.780570 8268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404c6e9ab8424368984521aaec9b08df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:26.783458 8502 tablet_bootstrap.cc:492] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270: Bootstrap starting.
I20260620 03:06:26.785182 8502 tablet_bootstrap.cc:654] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:26.788643 8504 tablet_bootstrap.cc:492] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f: Bootstrap starting.
I20260620 03:06:26.790171 8504 tablet_bootstrap.cc:654] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:26.793653 8503 tablet_bootstrap.cc:492] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748: Bootstrap starting.
I20260620 03:06:26.795087 8527 tablet_replica.cc:330] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270: stopping tablet replica
I20260620 03:06:26.795648 8504 tablet_bootstrap.cc:492] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f: No bootstrap required, opened a new log
I20260620 03:06:26.795629 8527 raft_consensus.cc:2229] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:06:26.795811 8504 ts_tablet_manager.cc:1397] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260620 03:06:26.796133 8527 raft_consensus.cc:2258] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:26.796355 8502 tablet_bootstrap.cc:492] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270: No bootstrap required, opened a new log
I20260620 03:06:26.796584 8502 ts_tablet_manager.cc:1397] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260620 03:06:26.796676 8504 raft_consensus.cc:348] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.796927 8504 raft_consensus.cc:374] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:26.797035 8504 raft_consensus.cc:729] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ebc275dcfa4053ad7231d7967a6c4f, State: Initialized, Role: FOLLOWER
I20260620 03:06:26.797343 8504 consensus_queue.cc:260] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.797901 8504 ts_tablet_manager.cc:1428] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 03:06:26.798290 8526 tablet_replica.cc:330] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748: stopping tablet replica
I20260620 03:06:26.798756 8526 raft_consensus.cc:2229] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:06:26.798812 8503 tablet_bootstrap.cc:654] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:26.797904 8502 raft_consensus.cc:348] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.798985 8502 raft_consensus.cc:374] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:26.799070 8502 raft_consensus.cc:729] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b0699d42404289801130a1359f1270, State: Initialized, Role: FOLLOWER
I20260620 03:06:26.799372 8502 consensus_queue.cc:260] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.800689 8502 ts_tablet_manager.cc:1428] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 03:06:26.801574 8526 raft_consensus.cc:2258] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:26.803587 8528 ts_tablet_manager.cc:1918] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:06:26.803617 8526 ts_tablet_manager.cc:1905] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:26.804953 8527 ts_tablet_manager.cc:1905] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:26.807490 8528 log.cc:1196] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/31f6298f4161462ebb3d8cd7df970761
I20260620 03:06:26.808049 8528 ts_tablet_manager.cc:1939] T 31f6298f4161462ebb3d8cd7df970761 P d3ebc275dcfa4053ad7231d7967a6c4f: Deleting consensus metadata
I20260620 03:06:26.808946 8503 tablet_bootstrap.cc:492] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748: No bootstrap required, opened a new log
I20260620 03:06:26.809157 8503 ts_tablet_manager.cc:1397] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.004s
I20260620 03:06:26.809841 7979 catalog_manager.cc:4841] TS d3ebc275dcfa4053ad7231d7967a6c4f (127.0.97.3:33765): tablet 31f6298f4161462ebb3d8cd7df970761 (table test-table [id=79d599e342d24bf5bf5076a9020ec386]) successfully deleted
I20260620 03:06:26.810243 8503 raft_consensus.cc:348] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.810537 8503 raft_consensus.cc:374] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:26.810688 8503 raft_consensus.cc:729] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4b81c6b987b42018936bd56b0862748, State: Initialized, Role: FOLLOWER
I20260620 03:06:26.811009 8503 consensus_queue.cc:260] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.811709 8526 ts_tablet_manager.cc:1918] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:06:26.811861 8526 log.cc:1196] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/31f6298f4161462ebb3d8cd7df970761
I20260620 03:06:26.812376 8526 ts_tablet_manager.cc:1939] T 31f6298f4161462ebb3d8cd7df970761 P b4b81c6b987b42018936bd56b0862748: Deleting consensus metadata
I20260620 03:06:26.814167 7977 catalog_manager.cc:4841] TS b4b81c6b987b42018936bd56b0862748 (127.0.97.2:37803): tablet 31f6298f4161462ebb3d8cd7df970761 (table test-table [id=79d599e342d24bf5bf5076a9020ec386]) successfully deleted
I20260620 03:06:26.814648 8503 ts_tablet_manager.cc:1428] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260620 03:06:26.814868 8527 ts_tablet_manager.cc:1918] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:06:26.815016 8527 log.cc:1196] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/31f6298f4161462ebb3d8cd7df970761
I20260620 03:06:26.815554 8527 ts_tablet_manager.cc:1939] T 31f6298f4161462ebb3d8cd7df970761 P 92b0699d42404289801130a1359f1270: Deleting consensus metadata
I20260620 03:06:26.817131 7978 catalog_manager.cc:4841] TS 92b0699d42404289801130a1359f1270 (127.0.97.1:46135): tablet 31f6298f4161462ebb3d8cd7df970761 (table test-table [id=79d599e342d24bf5bf5076a9020ec386]) successfully deleted
I20260620 03:06:26.832783 8510 raft_consensus.cc:482] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:06:26.833079 8510 raft_consensus.cc:504] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.834484 8510 leader_election.cc:288] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92b0699d42404289801130a1359f1270 (127.0.97.1:46135), b4b81c6b987b42018936bd56b0862748 (127.0.97.2:37803)
I20260620 03:06:26.838625 8509 raft_consensus.cc:482] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:06:26.838809 8509 raft_consensus.cc:504] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.840387 8509 leader_election.cc:288] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4b81c6b987b42018936bd56b0862748 (127.0.97.2:37803), d3ebc275dcfa4053ad7231d7967a6c4f (127.0.97.3:33765)
I20260620 03:06:26.856774 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.835083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48823 (local address 127.0.97.1:46135)
0620 03:06:26.835253 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:26.835260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:26.847791 (+ 12531us) server_negotiation.cc:407] Connection header received
0620 03:06:26.848573 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:26.848581 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:26.848682 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:26.848750 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:26.849406 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.850746 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.853598 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.855272 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.855407 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.855546 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:26.856209 (+ 663us) server_negotiation.cc:299] Negotiation successful
0620 03:06:26.856382 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 03:06:26.857781 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.834725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46135 (local address 127.0.97.3:48823)
0620 03:06:26.847684 (+ 12959us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:26.847706 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:26.848391 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:26.848990 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:26.848998 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:26.849029 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:26.849247 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.849261 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.850975 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.850982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.853406 (+ 2424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.853424 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.855737 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.855745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.855998 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.856049 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 03:06:26.857155 (+ 1106us) client_negotiation.cc:240] Negotiation successful
0620 03:06:26.857349 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12824,"thread_start_us":13080,"threads_started":1}
I20260620 03:06:26.858428 8150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404c6e9ab8424368984521aaec9b08df" candidate_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b0699d42404289801130a1359f1270" is_pre_election: true
I20260620 03:06:26.858717 8150 raft_consensus.cc:2454] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ebc275dcfa4053ad7231d7967a6c4f in term 0.
I20260620 03:06:26.859308 8360 leader_election.cc:302] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92b0699d42404289801130a1359f1270, d3ebc275dcfa4053ad7231d7967a6c4f; no voters:
I20260620 03:06:26.859764 8510 raft_consensus.cc:2781] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:06:26.859901 8510 raft_consensus.cc:482] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:06:26.860021 8510 raft_consensus.cc:3037] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:26.863375 8510 raft_consensus.cc:504] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.865051 8150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404c6e9ab8424368984521aaec9b08df" candidate_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b0699d42404289801130a1359f1270"
I20260620 03:06:26.865258 8150 raft_consensus.cc:3037] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:26.868342 8150 raft_consensus.cc:2454] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ebc275dcfa4053ad7231d7967a6c4f in term 1.
I20260620 03:06:26.868752 8360 leader_election.cc:302] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92b0699d42404289801130a1359f1270, d3ebc275dcfa4053ad7231d7967a6c4f; no voters:
I20260620 03:06:26.869735 8510 leader_election.cc:288] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [CANDIDATE]: Term 1 election: Requested vote from peers 92b0699d42404289801130a1359f1270 (127.0.97.1:46135), b4b81c6b987b42018936bd56b0862748 (127.0.97.2:37803)
I20260620 03:06:26.869956 8510 raft_consensus.cc:2781] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:26.870739 8510 raft_consensus.cc:686] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 LEADER]: Becoming Leader. State: Replica: d3ebc275dcfa4053ad7231d7967a6c4f, State: Running, Role: LEADER
I20260620 03:06:26.871136 8510 consensus_queue.cc:237] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.878744 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.848093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37489 (local address 127.0.97.3:33765)
0620 03:06:26.848265 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:26.848273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:26.864641 (+ 16368us) server_negotiation.cc:407] Connection header received
0620 03:06:26.864880 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:26.864886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:26.864953 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:26.865013 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:26.865869 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.866942 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.875994 (+ 9052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.877595 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.877698 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.877810 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:26.878350 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 03:06:26.878485 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 03:06:26.881006 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.840930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33765 (local address 127.0.97.1:37489)
0620 03:06:26.864550 (+ 23620us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:26.864570 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:26.864724 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:26.865189 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:26.865193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:26.865214 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:26.865669 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.865683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.867124 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.867130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.871520 (+ 4390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.871539 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.877998 (+ 6459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.878004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.878175 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.878207 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 03:06:26.878891 (+ 684us) client_negotiation.cc:240] Negotiation successful
0620 03:06:26.879086 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23494,"thread_start_us":23272,"threads_started":1}
I20260620 03:06:26.884016 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.835175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51835 (local address 127.0.97.2:37803)
0620 03:06:26.835445 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:26.835452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:26.868659 (+ 33207us) server_negotiation.cc:407] Connection header received
0620 03:06:26.869038 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:26.869045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:26.869111 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:26.869195 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:26.871705 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.873062 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.881301 (+ 8239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.882773 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.882884 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.882969 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:26.883543 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 03:06:26.883676 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 03:06:26.884697 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.834833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37803 (local address 127.0.97.3:51835)
0620 03:06:26.868561 (+ 33728us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:26.868578 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:26.868883 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:26.869536 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:26.869542 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:26.869564 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:26.870013 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.870027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.873281 (+ 3254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.873289 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.875779 (+ 2490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.875798 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.883144 (+ 7346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.883150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.883377 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.883408 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 03:06:26.884262 (+ 854us) client_negotiation.cc:240] Negotiation successful
0620 03:06:26.884411 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33609,"thread_start_us":33832,"threads_started":1}
I20260620 03:06:26.885504 8289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404c6e9ab8424368984521aaec9b08df" candidate_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4b81c6b987b42018936bd56b0862748" is_pre_election: true
I20260620 03:06:26.886026 8289 raft_consensus.cc:2454] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ebc275dcfa4053ad7231d7967a6c4f in term 0.
I20260620 03:06:26.886349 8428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404c6e9ab8424368984521aaec9b08df" candidate_uuid: "92b0699d42404289801130a1359f1270" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" is_pre_election: true
I20260620 03:06:26.885525 8288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404c6e9ab8424368984521aaec9b08df" candidate_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4b81c6b987b42018936bd56b0862748"
I20260620 03:06:26.886970 8288 raft_consensus.cc:3037] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:26.888635 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.840704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37803 (local address 127.0.97.1:33051)
0620 03:06:26.844651 (+ 3947us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:26.844676 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:26.844802 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:26.863770 (+ 18968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:26.863776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:26.863805 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:26.864160 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.864177 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.881777 (+ 17600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.881785 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.883707 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.883720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.887896 (+ 4176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.887904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.888076 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.888110 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 03:06:26.888174 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 03:06:26.888311 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3820,"thread_start_us":3897,"threads_started":1}
I20260620 03:06:26.889675 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.841117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33051 (local address 127.0.97.2:37803)
0620 03:06:26.863449 (+ 22332us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:26.863455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:26.863475 (+ 20us) server_negotiation.cc:407] Connection header received
0620 03:06:26.863537 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:26.863542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:26.863619 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:26.863758 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:26.877459 (+ 13701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.878756 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.883955 (+ 5199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.887682 (+ 3727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.888966 (+ 1284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.889116 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:26.889164 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 03:06:26.889303 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22133,"thread_start_us":22893,"threads_started":1}
I20260620 03:06:26.890421 8288 raft_consensus.cc:2454] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ebc275dcfa4053ad7231d7967a6c4f in term 1.
I20260620 03:06:26.890831 8289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404c6e9ab8424368984521aaec9b08df" candidate_uuid: "92b0699d42404289801130a1359f1270" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4b81c6b987b42018936bd56b0862748" is_pre_election: true
I20260620 03:06:26.891095 8289 raft_consensus.cc:2379] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92b0699d42404289801130a1359f1270 in current term 1: Already voted for candidate d3ebc275dcfa4053ad7231d7967a6c4f in this term.
I20260620 03:06:26.890800 7993 catalog_manager.cc:5495] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f reported cstate change: term changed from 0 to 1, leader changed from <none> to d3ebc275dcfa4053ad7231d7967a6c4f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } health_report { overall_health: HEALTHY } } }
I20260620 03:06:26.891834 8080 leader_election.cc:302] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92b0699d42404289801130a1359f1270; no voters: b4b81c6b987b42018936bd56b0862748, d3ebc275dcfa4053ad7231d7967a6c4f
I20260620 03:06:26.892222 8509 raft_consensus.cc:2726] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Jun 20 03:06:26 dist-test-slave-xv68 krb5kdc[7948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924784, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 03:06:26.946005 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.925872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36125 (local address 127.0.0.1:52114)
0620 03:06:26.926600 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:26.926616 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:26.926745 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:26.927165 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:26.927168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:26.927930 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:26.928279 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.928290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.929754 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.929757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.930702 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:26.930713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.931662 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.931667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:26.931792 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.937237 (+ 5445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:26.937255 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:26.939883 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:26.943563 (+ 3680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:26.943573 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:26.943580 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:26.944038 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:26.944428 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:26.944431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:26.944434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:26.944592 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:26.945188 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:26.945194 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:26.945351 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 03:06:26.945487 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 03:06:26.945685 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":422,"threads_started":1}
I20260620 03:06:26.946291 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:26.925878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52114 (local address 127.0.97.62:36125)
0620 03:06:26.926119 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:26.926125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:26.926713 (+ 588us) server_negotiation.cc:407] Connection header received
0620 03:06:26.926871 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:26.926877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:26.926954 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:26.927071 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:26.928425 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.929586 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.930851 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:26.931494 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:26.931619 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:26.940087 (+ 8468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:26.940095 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:26.940100 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:26.940141 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:26.943430 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:26.944153 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:26.944159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:26.944163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:26.944299 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:26.944704 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:26.944709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:26.944713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:26.945061 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:26.945209 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:26.945630 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 03:06:26.945852 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 03:06:26.951200 7993 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52114:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 03:06:26.952428 7993 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:06:26.959667 8130 tablet_service.cc:1467] Processing CreateTablet for tablet f608f48c747a47839380f6aa82c11d0c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33d06870f23344c2b3ef9b3a6966a1e9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 03:06:26.960283 8130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f608f48c747a47839380f6aa82c11d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:26.960040 8269 tablet_service.cc:1467] Processing CreateTablet for tablet f608f48c747a47839380f6aa82c11d0c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33d06870f23344c2b3ef9b3a6966a1e9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 03:06:26.960285 8407 tablet_service.cc:1467] Processing CreateTablet for tablet f608f48c747a47839380f6aa82c11d0c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33d06870f23344c2b3ef9b3a6966a1e9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 03:06:26.960624 8269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f608f48c747a47839380f6aa82c11d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:26.960778 8407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f608f48c747a47839380f6aa82c11d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:26.966169 8502 tablet_bootstrap.cc:492] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270: Bootstrap starting.
I20260620 03:06:26.967549 8503 tablet_bootstrap.cc:492] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748: Bootstrap starting.
I20260620 03:06:26.967859 8502 tablet_bootstrap.cc:654] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:26.968875 8504 tablet_bootstrap.cc:492] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f: Bootstrap starting.
I20260620 03:06:26.969111 8503 tablet_bootstrap.cc:654] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:26.970429 8504 tablet_bootstrap.cc:654] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:26.971520 8503 tablet_bootstrap.cc:492] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748: No bootstrap required, opened a new log
I20260620 03:06:26.971680 8503 ts_tablet_manager.cc:1397] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260620 03:06:26.972519 8503 raft_consensus.cc:348] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.972792 8503 raft_consensus.cc:374] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:26.972888 8503 raft_consensus.cc:729] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4b81c6b987b42018936bd56b0862748, State: Initialized, Role: FOLLOWER
I20260620 03:06:26.973124 8502 tablet_bootstrap.cc:492] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270: No bootstrap required, opened a new log
I20260620 03:06:26.973260 8502 ts_tablet_manager.cc:1397] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260620 03:06:26.973136 8503 consensus_queue.cc:260] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.973726 8503 ts_tablet_manager.cc:1428] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 03:06:26.974108 8513 tablet_replica.cc:439] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } }
I20260620 03:06:26.974431 8513 tablet_replica.cc:442] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:06:26.974545 8502 raft_consensus.cc:348] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.974751 8502 raft_consensus.cc:374] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:26.974820 8502 raft_consensus.cc:729] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b0699d42404289801130a1359f1270, State: Initialized, Role: FOLLOWER
I20260620 03:06:26.975054 8502 consensus_queue.cc:260] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.975531 8509 tablet_replica.cc:439] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } }
I20260620 03:06:26.975812 8509 tablet_replica.cc:442] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:06:26.975697 8502 ts_tablet_manager.cc:1428] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 03:06:26.978017 8504 tablet_bootstrap.cc:492] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f: No bootstrap required, opened a new log
I20260620 03:06:26.978193 8504 ts_tablet_manager.cc:1397] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260620 03:06:26.979300 8504 raft_consensus.cc:348] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.979576 8504 raft_consensus.cc:374] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:26.979671 8504 raft_consensus.cc:729] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ebc275dcfa4053ad7231d7967a6c4f, State: Initialized, Role: FOLLOWER
I20260620 03:06:26.979887 8504 consensus_queue.cc:260] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.980252 8510 tablet_replica.cc:439] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } }
I20260620 03:06:26.980427 8504 ts_tablet_manager.cc:1428] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 03:06:26.980554 8510 tablet_replica.cc:442] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:06:26.984165 8513 raft_consensus.cc:482] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:06:26.984352 8513 raft_consensus.cc:504] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.984962 8513 leader_election.cc:288] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92b0699d42404289801130a1359f1270 (127.0.97.1:46135), d3ebc275dcfa4053ad7231d7967a6c4f (127.0.97.3:33765)
I20260620 03:06:26.985357 8150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f608f48c747a47839380f6aa82c11d0c" candidate_uuid: "b4b81c6b987b42018936bd56b0862748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b0699d42404289801130a1359f1270" is_pre_election: true
I20260620 03:06:26.985396 8428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f608f48c747a47839380f6aa82c11d0c" candidate_uuid: "b4b81c6b987b42018936bd56b0862748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" is_pre_election: true
I20260620 03:06:26.985579 8150 raft_consensus.cc:2454] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4b81c6b987b42018936bd56b0862748 in term 0.
I20260620 03:06:26.985605 8428 raft_consensus.cc:2454] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4b81c6b987b42018936bd56b0862748 in term 0.
I20260620 03:06:26.985965 8221 leader_election.cc:302] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92b0699d42404289801130a1359f1270, b4b81c6b987b42018936bd56b0862748; no voters:
I20260620 03:06:26.986249 8513 raft_consensus.cc:2781] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:06:26.986373 8513 raft_consensus.cc:482] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:06:26.986483 8513 raft_consensus.cc:3037] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:26.989560 8513 raft_consensus.cc:504] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.990331 8513 leader_election.cc:288] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [CANDIDATE]: Term 1 election: Requested vote from peers 92b0699d42404289801130a1359f1270 (127.0.97.1:46135), d3ebc275dcfa4053ad7231d7967a6c4f (127.0.97.3:33765)
I20260620 03:06:26.990557 8428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f608f48c747a47839380f6aa82c11d0c" candidate_uuid: "b4b81c6b987b42018936bd56b0862748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f"
I20260620 03:06:26.990556 8150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f608f48c747a47839380f6aa82c11d0c" candidate_uuid: "b4b81c6b987b42018936bd56b0862748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b0699d42404289801130a1359f1270"
I20260620 03:06:26.990760 8428 raft_consensus.cc:3037] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:26.990762 8150 raft_consensus.cc:3037] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:26.992782 8428 raft_consensus.cc:2454] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4b81c6b987b42018936bd56b0862748 in term 1.
I20260620 03:06:26.993234 8222 leader_election.cc:302] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4b81c6b987b42018936bd56b0862748, d3ebc275dcfa4053ad7231d7967a6c4f; no voters:
I20260620 03:06:26.993551 8508 raft_consensus.cc:2781] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:26.993654 8150 raft_consensus.cc:2454] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4b81c6b987b42018936bd56b0862748 in term 1.
I20260620 03:06:26.993688 8508 raft_consensus.cc:686] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [term 1 LEADER]: Becoming Leader. State: Replica: b4b81c6b987b42018936bd56b0862748, State: Running, Role: LEADER
I20260620 03:06:26.993927 8508 consensus_queue.cc:237] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:26.995775 8521 tablet_replica.cc:439] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748: TxnStatusTablet state changed. Reason: New leader b4b81c6b987b42018936bd56b0862748. Latest consensus state: current_term: 1 leader_uuid: "b4b81c6b987b42018936bd56b0862748" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } }
I20260620 03:06:26.996173 8521 tablet_replica.cc:442] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748: This TxnStatusTablet replica's current role is: LEADER
I20260620 03:06:26.996807 8549 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 03:06:26.997037 8549 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 03:06:26.998000 7993 catalog_manager.cc:5495] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4b81c6b987b42018936bd56b0862748 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b4b81c6b987b42018936bd56b0862748" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } health_report { overall_health: UNKNOWN } } }
I20260620 03:06:27.012365 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:27.007575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44422 (local address 127.0.97.2:37803)
0620 03:06:27.007750 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:27.007758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:27.007811 (+ 53us) server_negotiation.cc:407] Connection header received
0620 03:06:27.008084 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:27.008092 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:27.008158 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:27.008251 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:06:27.008598 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.009437 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.010678 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.011080 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.011213 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:27.011675 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:06:27.011768 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:06:27.011863 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:27.012006 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 03:06:27.012075 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 03:06:27.012393 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:27.007512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37803 (local address 127.0.0.1:44422)
0620 03:06:27.007741 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:27.007762 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:27.007904 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:27.008274 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:27.008278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:27.008291 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 03:06:27.008474 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:27.008483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.009560 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.009567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:27.010538 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:27.010549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.011226 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.011232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:27.011438 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:27.011485 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 03:06:27.011870 (+ 385us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 03:06:27.011890 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 03:06:27.011986 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 03:06:27.012122 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 03:06:27.015770 8428 raft_consensus.cc:1261] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 FOLLOWER]: Refusing update from remote peer b4b81c6b987b42018936bd56b0862748: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:06:27.015817 8150 raft_consensus.cc:1261] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Refusing update from remote peer b4b81c6b987b42018936bd56b0862748: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:06:27.016356 8508 consensus_queue.cc:1035] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:27.016654 8521 consensus_queue.cc:1035] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:27.019275 8510 tablet_replica.cc:439] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f: TxnStatusTablet state changed. Reason: New leader b4b81c6b987b42018936bd56b0862748. Latest consensus state: current_term: 1 leader_uuid: "b4b81c6b987b42018936bd56b0862748" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } }
I20260620 03:06:27.019615 8510 tablet_replica.cc:442] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:06:27.021584 8508 tablet_replica.cc:439] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4b81c6b987b42018936bd56b0862748" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } }
I20260620 03:06:27.021884 8508 tablet_replica.cc:442] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748: This TxnStatusTablet replica's current role is: LEADER
I20260620 03:06:27.022431 8516 mvcc.cc:205] Tried to move back new op lower bound from 7298763927610535936 to 7298763927529496576. Current Snapshot: MvccSnapshot[applied={T|T < 7298763927610535936 or (T in {7298763927610535936})}]
I20260620 03:06:27.025808 8510 tablet_replica.cc:439] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4b81c6b987b42018936bd56b0862748" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } }
I20260620 03:06:27.026149 8510 tablet_replica.cc:442] T f608f48c747a47839380f6aa82c11d0c P d3ebc275dcfa4053ad7231d7967a6c4f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:06:27.029497 8509 tablet_replica.cc:439] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270: TxnStatusTablet state changed. Reason: New leader b4b81c6b987b42018936bd56b0862748. Latest consensus state: current_term: 1 leader_uuid: "b4b81c6b987b42018936bd56b0862748" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } }
I20260620 03:06:27.029780 8509 tablet_replica.cc:442] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:06:27.031898 8513 tablet_replica.cc:439] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4b81c6b987b42018936bd56b0862748" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } }
I20260620 03:06:27.032183 8513 tablet_replica.cc:442] T f608f48c747a47839380f6aa82c11d0c P b4b81c6b987b42018936bd56b0862748: This TxnStatusTablet replica's current role is: LEADER
I20260620 03:06:27.033337 8509 tablet_replica.cc:439] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4b81c6b987b42018936bd56b0862748" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } }
I20260620 03:06:27.033571 8509 tablet_replica.cc:442] T f608f48c747a47839380f6aa82c11d0c P 92b0699d42404289801130a1359f1270: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 03:06:27.037269 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:27.029272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35444 (local address 127.0.97.3:33765)
0620 03:06:27.029412 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:27.029418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:27.029435 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:06:27.029541 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:27.029547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:27.029635 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:27.029766 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:06:27.030281 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.031341 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.034441 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.035074 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.035201 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:27.035853 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:06:27.036538 (+ 685us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:06:27.036671 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:27.036746 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 03:06:27.036825 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 03:06:27.053300 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:27.046985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33777 (local address 127.0.97.2:37803)
0620 03:06:27.047197 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:27.047205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:27.047837 (+ 632us) server_negotiation.cc:407] Connection header received
0620 03:06:27.048026 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:27.048031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:27.048095 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:27.048173 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:06:27.048746 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.049578 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.051632 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.052099 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.052210 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:27.052709 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:06:27.052803 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:06:27.052866 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:27.053017 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 03:06:27.053086 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:06:27.053532 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:27.046976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37803 (local address 127.0.97.3:33777)
0620 03:06:27.047712 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:27.047733 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:27.047881 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:27.048216 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:27.048228 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:27.048240 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 03:06:27.048593 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:27.048607 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.049703 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.049708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:27.051491 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:27.051507 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.052293 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.052302 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:27.052477 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:27.052546 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 03:06:27.052901 (+ 355us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 03:06:27.052912 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 03:06:27.052990 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 03:06:27.053142 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":409,"threads_started":1}
I20260620 03:06:27.062942 8149 raft_consensus.cc:1261] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Refusing update from remote peer d3ebc275dcfa4053ad7231d7967a6c4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:06:27.063802 8289 raft_consensus.cc:1261] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748 [term 1 FOLLOWER]: Refusing update from remote peer d3ebc275dcfa4053ad7231d7967a6c4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:06:27.063648 8537 consensus_queue.cc:1035] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:27.064435 8537 consensus_queue.cc:1035] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:27.072010 8517 mvcc.cc:205] Tried to move back new op lower bound from 7298763927803842560 to 7298763927034544128. Current Snapshot: MvccSnapshot[applied={T|T < 7298763927803842560}]
I20260620 03:06:27.074059 8518 mvcc.cc:205] Tried to move back new op lower bound from 7298763927803842560 to 7298763927034544128. Current Snapshot: MvccSnapshot[applied={T|T < 7298763927803842560}]
I20260620 03:06:27.103873 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:27.097734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33281 (local address 127.0.97.3:33765)
0620 03:06:27.097930 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:27.097937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:27.098020 (+ 83us) server_negotiation.cc:407] Connection header received
0620 03:06:27.098200 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:27.098206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:27.098293 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:27.098388 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:06:27.098839 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.099939 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.101844 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.102536 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.102677 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:27.103074 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:06:27.103212 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:06:27.103340 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:27.103512 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 03:06:27.103590 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 03:06:27.104015 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:27.097200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33765 (local address 127.0.97.2:33281)
0620 03:06:27.097947 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:27.097963 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:27.098074 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:27.098403 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:27.098406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:27.098414 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 03:06:27.098696 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:27.098703 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.100320 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.100326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:27.101709 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:27.101721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.102706 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.102710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:27.102833 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:27.102881 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 03:06:27.103375 (+ 494us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 03:06:27.103386 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 03:06:27.103475 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 03:06:27.103605 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":373,"threads_started":1}
I20260620 03:06:27.157135 7992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:06:27.157403 7992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:06:27.160799 7992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7: Sending DeleteTablet for 3 replicas of tablet 404c6e9ab8424368984521aaec9b08df
I20260620 03:06:27.161579 8130 tablet_service.cc:1514] Processing DeleteTablet for tablet 404c6e9ab8424368984521aaec9b08df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:27 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60566
I20260620 03:06:27.161854 8527 tablet_replica.cc:330] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270: stopping tablet replica
I20260620 03:06:27.161983 8407 tablet_service.cc:1514] Processing DeleteTablet for tablet 404c6e9ab8424368984521aaec9b08df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:27 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35428
I20260620 03:06:27.162046 8527 raft_consensus.cc:2229] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:06:27.162243 8527 raft_consensus.cc:2258] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:27.162263 8528 tablet_replica.cc:330] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f: stopping tablet replica
I20260620 03:06:27.162451 8528 raft_consensus.cc:2229] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:06:27.162927 8528 raft_consensus.cc:2258] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:27.163615 8269 tablet_service.cc:1514] Processing DeleteTablet for tablet 404c6e9ab8424368984521aaec9b08df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:27 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44402
I20260620 03:06:27.163682 8527 ts_tablet_manager.cc:1905] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:27.163890 8526 tablet_replica.cc:330] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748: stopping tablet replica
I20260620 03:06:27.164091 8526 raft_consensus.cc:2229] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:06:27.164281 8526 raft_consensus.cc:2258] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:27.165016 8528 ts_tablet_manager.cc:1905] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:27.167891 8526 ts_tablet_manager.cc:1905] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:27.171940 8528 ts_tablet_manager.cc:1918] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 03:06:27.175443 8526 ts_tablet_manager.cc:1918] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 03:06:27.175596 8526 log.cc:1196] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/404c6e9ab8424368984521aaec9b08df
I20260620 03:06:27.176013 8526 ts_tablet_manager.cc:1939] T 404c6e9ab8424368984521aaec9b08df P b4b81c6b987b42018936bd56b0862748: Deleting consensus metadata
I20260620 03:06:27.177160 7977 catalog_manager.cc:4841] TS b4b81c6b987b42018936bd56b0862748 (127.0.97.2:37803): tablet 404c6e9ab8424368984521aaec9b08df (table test-table [id=5b31f1fbdbad49c5b06b5c868dca64b9]) successfully deleted
I20260620 03:06:27.178956 8527 ts_tablet_manager.cc:1918] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 03:06:27.179100 8527 log.cc:1196] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/404c6e9ab8424368984521aaec9b08df
I20260620 03:06:27.179420 8528 log.cc:1196] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/404c6e9ab8424368984521aaec9b08df
I20260620 03:06:27.179590 8527 ts_tablet_manager.cc:1939] T 404c6e9ab8424368984521aaec9b08df P 92b0699d42404289801130a1359f1270: Deleting consensus metadata
I20260620 03:06:27.179863 8528 ts_tablet_manager.cc:1939] T 404c6e9ab8424368984521aaec9b08df P d3ebc275dcfa4053ad7231d7967a6c4f: Deleting consensus metadata
I20260620 03:06:27.180637 7978 catalog_manager.cc:4841] TS 92b0699d42404289801130a1359f1270 (127.0.97.1:46135): tablet 404c6e9ab8424368984521aaec9b08df (table test-table [id=5b31f1fbdbad49c5b06b5c868dca64b9]) successfully deleted
I20260620 03:06:27.180958 7979 catalog_manager.cc:4841] TS d3ebc275dcfa4053ad7231d7967a6c4f (127.0.97.3:33765): tablet 404c6e9ab8424368984521aaec9b08df (table test-table [id=5b31f1fbdbad49c5b06b5c868dca64b9]) successfully deleted
Jun 20 03:06:27 dist-test-slave-xv68 krb5kdc[7948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 03:06:27 dist-test-slave-xv68 krb5kdc[7948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:27 dist-test-slave-xv68 krb5kdc[7948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 03:06:27.242529 8088 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:27.229406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.0.97.62:36125)
0620 03:06:27.229591 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:27.229597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:27.230114 (+ 517us) server_negotiation.cc:407] Connection header received
0620 03:06:27.230317 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:27.230324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:27.230415 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:27.230534 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:27.232165 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.233311 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.235171 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.235862 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.236010 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:27.242254 (+ 6244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52118: BlockingRecv error: recv got EOF from 127.0.0.1:52118 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 03:06:27 dist-test-slave-xv68 krb5kdc[7948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 03:06:27.264756 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:27.246148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.0.97.62:36125)
0620 03:06:27.246307 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:27.246312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:27.247744 (+ 1432us) server_negotiation.cc:407] Connection header received
0620 03:06:27.247911 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:27.247917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:27.247995 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:27.248100 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:27.249647 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.250737 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.252612 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.253239 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.253354 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:27.258072 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:27.258079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:27.258084 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:27.258127 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:27.261744 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:27.262579 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:27.262585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:27.262588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:27.262738 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:27.263142 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:27.263146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:27.263150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:27.263546 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:27.263789 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:27.264118 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 03:06:27.264352 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 03:06:27.268646 7992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:06:27.269371 7992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:06:27.276785 8130 tablet_service.cc:1467] Processing CreateTablet for tablet 57bc399d85da46b094caedc72fdc7146 (DEFAULT_TABLE table=test-table [id=4991b26ab2e94b8394db868feab2f2f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:27.277211 8130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57bc399d85da46b094caedc72fdc7146. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:27.277141 8269 tablet_service.cc:1467] Processing CreateTablet for tablet 57bc399d85da46b094caedc72fdc7146 (DEFAULT_TABLE table=test-table [id=4991b26ab2e94b8394db868feab2f2f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:27.277539 8269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57bc399d85da46b094caedc72fdc7146. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:27.283568 8502 tablet_bootstrap.cc:492] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270: Bootstrap starting.
I20260620 03:06:27.284090 8407 tablet_service.cc:1467] Processing CreateTablet for tablet 57bc399d85da46b094caedc72fdc7146 (DEFAULT_TABLE table=test-table [id=4991b26ab2e94b8394db868feab2f2f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:27.284502 8407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57bc399d85da46b094caedc72fdc7146. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:27.285142 8502 tablet_bootstrap.cc:654] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:27.295673 8502 tablet_bootstrap.cc:492] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270: No bootstrap required, opened a new log
I20260620 03:06:27.295868 8502 ts_tablet_manager.cc:1397] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.007s
I20260620 03:06:27.296084 8503 tablet_bootstrap.cc:492] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748: Bootstrap starting.
I20260620 03:06:27.296703 8502 raft_consensus.cc:348] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:27.297043 8502 raft_consensus.cc:374] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:27.297158 8502 raft_consensus.cc:729] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b0699d42404289801130a1359f1270, State: Initialized, Role: FOLLOWER
I20260620 03:06:27.297437 8502 consensus_queue.cc:260] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:27.297755 8503 tablet_bootstrap.cc:654] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:27.298151 8502 ts_tablet_manager.cc:1428] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 03:06:27.301548 8504 tablet_bootstrap.cc:492] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f: Bootstrap starting.
I20260620 03:06:27.303188 8504 tablet_bootstrap.cc:654] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:27.304728 8503 tablet_bootstrap.cc:492] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748: No bootstrap required, opened a new log
I20260620 03:06:27.304873 8503 ts_tablet_manager.cc:1397] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20260620 03:06:27.305616 8503 raft_consensus.cc:348] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:27.305887 8503 raft_consensus.cc:374] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:27.305984 8503 raft_consensus.cc:729] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4b81c6b987b42018936bd56b0862748, State: Initialized, Role: FOLLOWER
I20260620 03:06:27.306202 8503 consensus_queue.cc:260] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:27.307070 8503 ts_tablet_manager.cc:1428] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 03:06:27.312721 8504 tablet_bootstrap.cc:492] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f: No bootstrap required, opened a new log
I20260620 03:06:27.312881 8504 ts_tablet_manager.cc:1397] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20260620 03:06:27.313612 8504 raft_consensus.cc:348] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:27.313875 8504 raft_consensus.cc:374] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:27.313982 8504 raft_consensus.cc:729] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ebc275dcfa4053ad7231d7967a6c4f, State: Initialized, Role: FOLLOWER
I20260620 03:06:27.314428 8504 consensus_queue.cc:260] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:27.315547 8504 ts_tablet_manager.cc:1428] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 03:06:27.620491 8560 raft_consensus.cc:482] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:06:27.620728 8560 raft_consensus.cc:504] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:27.621556 8560 leader_election.cc:288] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92b0699d42404289801130a1359f1270 (127.0.97.1:46135), b4b81c6b987b42018936bd56b0862748 (127.0.97.2:37803)
I20260620 03:06:27.622191 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bc399d85da46b094caedc72fdc7146" candidate_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b0699d42404289801130a1359f1270" is_pre_election: true
I20260620 03:06:27.622282 8288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bc399d85da46b094caedc72fdc7146" candidate_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4b81c6b987b42018936bd56b0862748" is_pre_election: true
I20260620 03:06:27.622447 8149 raft_consensus.cc:2454] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ebc275dcfa4053ad7231d7967a6c4f in term 0.
I20260620 03:06:27.622565 8288 raft_consensus.cc:2454] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ebc275dcfa4053ad7231d7967a6c4f in term 0.
I20260620 03:06:27.623067 8359 leader_election.cc:302] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4b81c6b987b42018936bd56b0862748, d3ebc275dcfa4053ad7231d7967a6c4f; no voters:
I20260620 03:06:27.623417 8560 raft_consensus.cc:2781] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:06:27.623540 8560 raft_consensus.cc:482] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:06:27.623632 8560 raft_consensus.cc:3037] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:27.626912 8560 raft_consensus.cc:504] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:27.628118 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bc399d85da46b094caedc72fdc7146" candidate_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b0699d42404289801130a1359f1270"
I20260620 03:06:27.628111 8288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bc399d85da46b094caedc72fdc7146" candidate_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4b81c6b987b42018936bd56b0862748"
I20260620 03:06:27.628333 8149 raft_consensus.cc:3037] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:27.628336 8288 raft_consensus.cc:3037] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:27.631480 8288 raft_consensus.cc:2454] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ebc275dcfa4053ad7231d7967a6c4f in term 1.
I20260620 03:06:27.631207 8149 raft_consensus.cc:2454] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ebc275dcfa4053ad7231d7967a6c4f in term 1.
I20260620 03:06:27.631942 8359 leader_election.cc:302] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4b81c6b987b42018936bd56b0862748, d3ebc275dcfa4053ad7231d7967a6c4f; no voters:
I20260620 03:06:27.632256 8537 raft_consensus.cc:2781] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:27.632695 8537 raft_consensus.cc:686] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 LEADER]: Becoming Leader. State: Replica: d3ebc275dcfa4053ad7231d7967a6c4f, State: Running, Role: LEADER
I20260620 03:06:27.633031 8537 consensus_queue.cc:237] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:27.634863 8560 leader_election.cc:288] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [CANDIDATE]: Term 1 election: Requested vote from peers 92b0699d42404289801130a1359f1270 (127.0.97.1:46135), b4b81c6b987b42018936bd56b0862748 (127.0.97.2:37803)
I20260620 03:06:27.645195 7993 catalog_manager.cc:5495] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f reported cstate change: term changed from 0 to 1, leader changed from <none> to d3ebc275dcfa4053ad7231d7967a6c4f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } health_report { overall_health: HEALTHY } } }
I20260620 03:06:27.670208 8581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:27.662706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.0.97.3:33765)
0620 03:06:27.663797 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:27.663803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:27.663823 (+ 20us) server_negotiation.cc:407] Connection header received
0620 03:06:27.663888 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:27.663893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:27.663976 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:27.664480 (+ 504us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:06:27.664702 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.665843 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.667864 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:27.668605 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:27.668741 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:27.669247 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:06:27.669361 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:06:27.669448 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:27.669677 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 03:06:27.669749 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944,"thread_start_us":757,"threads_started":1}
I20260620 03:06:27.673514 8149 raft_consensus.cc:1261] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Refusing update from remote peer d3ebc275dcfa4053ad7231d7967a6c4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:06:27.673652 8288 raft_consensus.cc:1261] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748 [term 1 FOLLOWER]: Refusing update from remote peer d3ebc275dcfa4053ad7231d7967a6c4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:06:27.674284 8560 consensus_queue.cc:1035] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:27.674827 8578 consensus_queue.cc:1035] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:27.703215 7993 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:06:27.703501 7993 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:06:27.707625 7993 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7: Sending DeleteTablet for 3 replicas of tablet 57bc399d85da46b094caedc72fdc7146
I20260620 03:06:27.708855 8130 tablet_service.cc:1514] Processing DeleteTablet for tablet 57bc399d85da46b094caedc72fdc7146 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:27 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60566
I20260620 03:06:27.709228 8269 tablet_service.cc:1514] Processing DeleteTablet for tablet 57bc399d85da46b094caedc72fdc7146 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:27 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44402
I20260620 03:06:27.709931 8407 tablet_service.cc:1514] Processing DeleteTablet for tablet 57bc399d85da46b094caedc72fdc7146 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:27 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35428
I20260620 03:06:27.713899 7992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:06:27.714534 7992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:06:27.715726 8590 tablet_replica.cc:330] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270: stopping tablet replica
I20260620 03:06:27.716557 8591 tablet_replica.cc:330] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748: stopping tablet replica
I20260620 03:06:27.716811 8591 raft_consensus.cc:2229] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:06:27.717147 8591 raft_consensus.cc:2258] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:27.717730 8592 tablet_replica.cc:330] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f: stopping tablet replica
I20260620 03:06:27.717957 8592 raft_consensus.cc:2229] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:06:27.718281 8592 raft_consensus.cc:2258] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:27.719450 8591 ts_tablet_manager.cc:1905] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:27.722911 8130 tablet_service.cc:1467] Processing CreateTablet for tablet 21acbd5337534981baee476152e5b96c (DEFAULT_TABLE table=test-table [id=6d871a0a47b4436eb2e620cf5d16408a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:27.723335 8130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21acbd5337534981baee476152e5b96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:27.724776 8592 ts_tablet_manager.cc:1905] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:27.727628 8269 tablet_service.cc:1467] Processing CreateTablet for tablet 21acbd5337534981baee476152e5b96c (DEFAULT_TABLE table=test-table [id=6d871a0a47b4436eb2e620cf5d16408a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:27.728016 8269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21acbd5337534981baee476152e5b96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:27.728371 8590 raft_consensus.cc:2229] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:06:27.728973 8590 raft_consensus.cc:2258] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:27.733063 8408 tablet_service.cc:1467] Processing CreateTablet for tablet 21acbd5337534981baee476152e5b96c (DEFAULT_TABLE table=test-table [id=6d871a0a47b4436eb2e620cf5d16408a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:27.733513 8408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21acbd5337534981baee476152e5b96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:27.735152 8590 ts_tablet_manager.cc:1905] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:27.741729 8590 ts_tablet_manager.cc:1918] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:06:27.741861 8590 log.cc:1196] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/57bc399d85da46b094caedc72fdc7146
I20260620 03:06:27.742268 8590 ts_tablet_manager.cc:1939] T 57bc399d85da46b094caedc72fdc7146 P 92b0699d42404289801130a1359f1270: Deleting consensus metadata
I20260620 03:06:27.742272 8591 ts_tablet_manager.cc:1918] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:06:27.742406 8591 log.cc:1196] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/57bc399d85da46b094caedc72fdc7146
I20260620 03:06:27.742897 8591 ts_tablet_manager.cc:1939] T 57bc399d85da46b094caedc72fdc7146 P b4b81c6b987b42018936bd56b0862748: Deleting consensus metadata
I20260620 03:06:27.743477 7978 catalog_manager.cc:4841] TS 92b0699d42404289801130a1359f1270 (127.0.97.1:46135): tablet 57bc399d85da46b094caedc72fdc7146 (table test-table [id=4991b26ab2e94b8394db868feab2f2f8]) successfully deleted
I20260620 03:06:27.743630 8592 ts_tablet_manager.cc:1918] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:06:27.743763 8592 log.cc:1196] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/57bc399d85da46b094caedc72fdc7146
I20260620 03:06:27.743943 7977 catalog_manager.cc:4841] TS b4b81c6b987b42018936bd56b0862748 (127.0.97.2:37803): tablet 57bc399d85da46b094caedc72fdc7146 (table test-table [id=4991b26ab2e94b8394db868feab2f2f8]) successfully deleted
I20260620 03:06:27.744266 8592 ts_tablet_manager.cc:1939] T 57bc399d85da46b094caedc72fdc7146 P d3ebc275dcfa4053ad7231d7967a6c4f: Deleting consensus metadata
I20260620 03:06:27.745280 7979 catalog_manager.cc:4841] TS d3ebc275dcfa4053ad7231d7967a6c4f (127.0.97.3:33765): tablet 57bc399d85da46b094caedc72fdc7146 (table test-table [id=4991b26ab2e94b8394db868feab2f2f8]) successfully deleted
I20260620 03:06:27.746119 8504 tablet_bootstrap.cc:492] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f: Bootstrap starting.
I20260620 03:06:27.746210 8502 tablet_bootstrap.cc:492] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270: Bootstrap starting.
I20260620 03:06:27.747738 8504 tablet_bootstrap.cc:654] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:27.747750 8502 tablet_bootstrap.cc:654] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:27.748370 8503 tablet_bootstrap.cc:492] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748: Bootstrap starting.
I20260620 03:06:27.749979 8503 tablet_bootstrap.cc:654] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:27.760295 8502 tablet_bootstrap.cc:492] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270: No bootstrap required, opened a new log
I20260620 03:06:27.760370 8504 tablet_bootstrap.cc:492] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f: No bootstrap required, opened a new log
I20260620 03:06:27.760456 8502 ts_tablet_manager.cc:1397] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.002s
I20260620 03:06:27.760550 8504 ts_tablet_manager.cc:1397] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260620 03:06:27.761198 8502 raft_consensus.cc:348] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:27.761200 8504 raft_consensus.cc:348] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:27.761448 8502 raft_consensus.cc:374] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:27.761466 8504 raft_consensus.cc:374] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:27.761518 8502 raft_consensus.cc:729] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b0699d42404289801130a1359f1270, State: Initialized, Role: FOLLOWER
I20260620 03:06:27.761547 8504 raft_consensus.cc:729] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ebc275dcfa4053ad7231d7967a6c4f, State: Initialized, Role: FOLLOWER
I20260620 03:06:27.761790 8504 consensus_queue.cc:260] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:27.761821 8502 consensus_queue.cc:260] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:27.762270 8504 ts_tablet_manager.cc:1428] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 03:06:27.763078 8503 tablet_bootstrap.cc:492] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748: No bootstrap required, opened a new log
I20260620 03:06:27.763270 8503 ts_tablet_manager.cc:1397] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.002s
I20260620 03:06:27.764073 8503 raft_consensus.cc:348] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:27.764317 8503 raft_consensus.cc:374] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:27.764418 8503 raft_consensus.cc:729] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4b81c6b987b42018936bd56b0862748, State: Initialized, Role: FOLLOWER
I20260620 03:06:27.764659 8503 consensus_queue.cc:260] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:27.766172 8503 ts_tablet_manager.cc:1428] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 03:06:27.766808 8502 ts_tablet_manager.cc:1428] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260620 03:06:28.040004 8508 raft_consensus.cc:482] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:06:28.040215 8508 raft_consensus.cc:504] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:28.040920 8508 leader_election.cc:288] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92b0699d42404289801130a1359f1270 (127.0.97.1:46135), d3ebc275dcfa4053ad7231d7967a6c4f (127.0.97.3:33765)
I20260620 03:06:28.041424 8428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21acbd5337534981baee476152e5b96c" candidate_uuid: "b4b81c6b987b42018936bd56b0862748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" is_pre_election: true
I20260620 03:06:28.041422 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21acbd5337534981baee476152e5b96c" candidate_uuid: "b4b81c6b987b42018936bd56b0862748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b0699d42404289801130a1359f1270" is_pre_election: true
I20260620 03:06:28.041678 8149 raft_consensus.cc:2454] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4b81c6b987b42018936bd56b0862748 in term 0.
I20260620 03:06:28.041685 8428 raft_consensus.cc:2454] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4b81c6b987b42018936bd56b0862748 in term 0.
I20260620 03:06:28.042146 8221 leader_election.cc:302] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92b0699d42404289801130a1359f1270, b4b81c6b987b42018936bd56b0862748; no voters:
I20260620 03:06:28.042419 8508 raft_consensus.cc:2781] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:06:28.042559 8508 raft_consensus.cc:482] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:06:28.042655 8508 raft_consensus.cc:3037] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:28.044858 8508 raft_consensus.cc:504] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:28.045270 8508 leader_election.cc:288] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [CANDIDATE]: Term 1 election: Requested vote from peers 92b0699d42404289801130a1359f1270 (127.0.97.1:46135), d3ebc275dcfa4053ad7231d7967a6c4f (127.0.97.3:33765)
I20260620 03:06:28.045689 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21acbd5337534981baee476152e5b96c" candidate_uuid: "b4b81c6b987b42018936bd56b0862748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b0699d42404289801130a1359f1270"
I20260620 03:06:28.045737 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21acbd5337534981baee476152e5b96c" candidate_uuid: "b4b81c6b987b42018936bd56b0862748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f"
I20260620 03:06:28.045905 8149 raft_consensus.cc:3037] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:28.045923 8427 raft_consensus.cc:3037] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:28.048784 8149 raft_consensus.cc:2454] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4b81c6b987b42018936bd56b0862748 in term 1.
I20260620 03:06:28.048799 8427 raft_consensus.cc:2454] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4b81c6b987b42018936bd56b0862748 in term 1.
I20260620 03:06:28.049196 8221 leader_election.cc:302] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 92b0699d42404289801130a1359f1270, b4b81c6b987b42018936bd56b0862748, d3ebc275dcfa4053ad7231d7967a6c4f; no voters:
I20260620 03:06:28.049551 8508 raft_consensus.cc:2781] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:28.049762 8508 raft_consensus.cc:686] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [term 1 LEADER]: Becoming Leader. State: Replica: b4b81c6b987b42018936bd56b0862748, State: Running, Role: LEADER
I20260620 03:06:28.050017 8508 consensus_queue.cc:237] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } }
I20260620 03:06:28.052465 7992 catalog_manager.cc:5495] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4b81c6b987b42018936bd56b0862748 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b4b81c6b987b42018936bd56b0862748" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4b81c6b987b42018936bd56b0862748" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } health_report { overall_health: UNKNOWN } } }
I20260620 03:06:28.126959 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:28.122372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48966 (local address 127.0.97.2:37803)
0620 03:06:28.122875 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:28.122879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:28.122897 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:06:28.122949 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:28.122953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:28.123008 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:28.123087 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:06:28.123380 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:28.124070 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:28.125349 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:28.125752 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:28.125843 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:28.126202 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:06:28.126297 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:06:28.126358 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:28.126537 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 03:06:28.126589 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":267,"threads_started":1}
I20260620 03:06:28.139037 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:28.134330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59585 (local address 127.0.97.2:37803)
0620 03:06:28.134493 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:28.134497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:28.134991 (+ 494us) server_negotiation.cc:407] Connection header received
0620 03:06:28.135133 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:28.135136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:28.135212 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:28.135346 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:06:28.135766 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:28.136472 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:28.137698 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:28.138082 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:28.138174 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:28.138497 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:06:28.138578 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:06:28.138639 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:28.138755 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 03:06:28.138833 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":61}
I20260620 03:06:28.139182 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:28.134198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37803 (local address 127.0.97.2:59585)
0620 03:06:28.134862 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:28.134878 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:28.135025 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:28.135371 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:28.135375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:28.135386 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 03:06:28.135633 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:28.135640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:28.136586 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:28.136589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:28.137575 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:28.137585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:28.138229 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:28.138233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:28.138345 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:28.138376 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 03:06:28.138653 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 03:06:28.138661 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 03:06:28.138721 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 03:06:28.138827 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":2,"thread_start_us":414,"threads_started":1}
I20260620 03:06:28.146469 8427 raft_consensus.cc:1261] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 FOLLOWER]: Refusing update from remote peer b4b81c6b987b42018936bd56b0862748: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:06:28.146595 8149 raft_consensus.cc:1261] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Refusing update from remote peer b4b81c6b987b42018936bd56b0862748: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:06:28.147099 8596 consensus_queue.cc:1035] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b0699d42404289801130a1359f1270" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:28.147399 8598 consensus_queue.cc:1035] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ebc275dcfa4053ad7231d7967a6c4f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:06:28.187922 7991 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:06:28.188153 7991 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:06:28.190834 7991 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1f760ed552844808735fcfad9de87c7: Sending DeleteTablet for 3 replicas of tablet 21acbd5337534981baee476152e5b96c
I20260620 03:06:28.191565 8130 tablet_service.cc:1514] Processing DeleteTablet for tablet 21acbd5337534981baee476152e5b96c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:28 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60566
I20260620 03:06:28.191596 8269 tablet_service.cc:1514] Processing DeleteTablet for tablet 21acbd5337534981baee476152e5b96c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:28 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44402
I20260620 03:06:28.191856 8590 tablet_replica.cc:330] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270: stopping tablet replica
I20260620 03:06:28.191957 8408 tablet_service.cc:1514] Processing DeleteTablet for tablet 21acbd5337534981baee476152e5b96c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:06:28 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35428
I20260620 03:06:28.192036 8590 raft_consensus.cc:2229] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:06:28.192217 8592 tablet_replica.cc:330] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f: stopping tablet replica
I20260620 03:06:28.192216 8590 raft_consensus.cc:2258] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:28.192265 8591 tablet_replica.cc:330] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748: stopping tablet replica
I20260620 03:06:28.192380 8592 raft_consensus.cc:2229] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:06:28.192451 8591 raft_consensus.cc:2229] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:06:28.192549 8592 raft_consensus.cc:2258] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:28.192749 8591 raft_consensus.cc:2258] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:06:28.192862 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 8064
I20260620 03:06:28.193496 8590 ts_tablet_manager.cc:1905] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:28.193733 8592 ts_tablet_manager.cc:1905] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:28.194590 8591 ts_tablet_manager.cc:1905] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:06:28.199999 8590 ts_tablet_manager.cc:1918] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 03:06:28.200141 8590 log.cc:1196] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/21acbd5337534981baee476152e5b96c
I20260620 03:06:28.200551 8590 ts_tablet_manager.cc:1939] T 21acbd5337534981baee476152e5b96c P 92b0699d42404289801130a1359f1270: Deleting consensus metadata
I20260620 03:06:28.200925 8591 ts_tablet_manager.cc:1918] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 03:06:28.201062 8591 log.cc:1196] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/21acbd5337534981baee476152e5b96c
I20260620 03:06:28.201478 8591 ts_tablet_manager.cc:1939] T 21acbd5337534981baee476152e5b96c P b4b81c6b987b42018936bd56b0862748: Deleting consensus metadata
I20260620 03:06:28.201603 7978 catalog_manager.cc:4841] TS 92b0699d42404289801130a1359f1270 (127.0.97.1:46135): tablet 21acbd5337534981baee476152e5b96c (table test-table [id=6d871a0a47b4436eb2e620cf5d16408a]) successfully deleted
I20260620 03:06:28.201802 8592 ts_tablet_manager.cc:1918] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 03:06:28.201958 8592 log.cc:1196] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/21acbd5337534981baee476152e5b96c
I20260620 03:06:28.202333 8592 ts_tablet_manager.cc:1939] T 21acbd5337534981baee476152e5b96c P d3ebc275dcfa4053ad7231d7967a6c4f: Deleting consensus metadata
I20260620 03:06:28.202508 7977 catalog_manager.cc:4841] TS b4b81c6b987b42018936bd56b0862748 (127.0.97.2:37803): tablet 21acbd5337534981baee476152e5b96c (table test-table [id=6d871a0a47b4436eb2e620cf5d16408a]) successfully deleted
I20260620 03:06:28.203725 7979 catalog_manager.cc:4841] TS d3ebc275dcfa4053ad7231d7967a6c4f (127.0.97.3:33765): tablet 21acbd5337534981baee476152e5b96c (table test-table [id=6d871a0a47b4436eb2e620cf5d16408a]) successfully deleted
Jun 20 03:06:28 dist-test-slave-xv68 krb5kdc[7948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260620 03:06:28.216279 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:28.202540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.0.97.1:46135)
0620 03:06:28.202960 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:28.202965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:28.202980 (+ 15us) server_negotiation.cc:407] Connection header received
0620 03:06:28.203022 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:28.203025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:28.203078 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:28.203200 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:28.204547 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:28.205271 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:28.206642 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:28.207072 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:28.207241 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:28.211048 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:28.211060 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:28.211076 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:28.211104 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:28.213811 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:28.214400 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:28.214408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:28.214410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:28.214522 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:28.214962 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:28.214966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:28.214968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:28.215348 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:28.215570 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:28.215847 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 03:06:28.216018 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":270,"threads_started":1}
I20260620 03:06:28.216910 8195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:28.315270 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 8064
I20260620 03:06:28.333478 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 8204
Jun 20 03:06:28 dist-test-slave-xv68 krb5kdc[7948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260620 03:06:28.347837 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:28.334145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.0.97.2:37803)
0620 03:06:28.334364 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:28.334370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:28.334389 (+ 19us) server_negotiation.cc:407] Connection header received
0620 03:06:28.334582 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:28.334588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:28.334659 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:28.334763 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:28.336191 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:28.336967 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:28.338175 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:28.338589 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:28.338718 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:28.342834 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:28.342850 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:28.342866 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:28.342901 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:28.345594 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:28.346141 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:28.346148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:28.346150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:28.346261 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:28.346584 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:28.346588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:28.346591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:28.346930 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:28.347171 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:28.347432 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 03:06:28.347587 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:06:28.348416 8334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:28.500167 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 8204
I20260620 03:06:28.525322 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 8343
Jun 20 03:06:28 dist-test-slave-xv68 krb5kdc[7948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260620 03:06:28.539978 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:28.526063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.0.97.3:33765)
0620 03:06:28.526541 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:28.526546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:28.526561 (+ 15us) server_negotiation.cc:407] Connection header received
0620 03:06:28.526611 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:28.526615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:28.526670 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:28.526811 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:28.528137 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:28.528825 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:28.530038 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:28.530428 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:28.530519 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:28.534700 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:28.534715 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:28.534731 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:28.534763 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:28.537626 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:28.538198 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:28.538204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:28.538206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:28.538323 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:28.538619 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:28.538622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:28.538624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:28.538928 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:28.539127 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:28.539454 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 03:06:28.539607 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":265,"threads_started":1}
I20260620 03:06:28.540648 8473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:28.661543 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 8343
I20260620 03:06:28.683120 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 7961
I20260620 03:06:28.684340 8033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:28.798173 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 7961
2026-06-20T03:06:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5830 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:06:28 dist-test-slave-xv68 krb5kdc[8616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:06:28 dist-test-slave-xv68 krb5kdc[8616](info): set up 2 sockets
Jun 20 03:06:28 dist-test-slave-xv68 krb5kdc[8616](info): commencing operation
krb5kdc: starting...
W20260620 03:06:29.892282 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:06:30 dist-test-slave-xv68 krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:06:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:06:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:06:30.106272 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45143
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:30.425371 8629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:30.425709 8629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:30.425886 8629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:30.432904 8629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:06:30.433061 8629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:30.433176 8629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:30.433280 8629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:06:30.433354 8629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:06:30.442988 8629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:30.444339 8629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:30.445749 8629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:30.457154 8635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:30.457746 8637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:30.457774 8634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:30.458066 8629 server_base.cc:1034] running on GCE node
I20260620 03:06:30.458823 8629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:30.460212 8629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:30.461441 8629 hybrid_clock.cc:648] HybridClock initialized: now 1781924790461399 us; error 36 us; skew 500 ppm
I20260620 03:06:30.461773 8629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T03:06:30Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1732 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:06:30 dist-test-slave-xv68 krb5kdc[8643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:06:30 dist-test-slave-xv68 krb5kdc[8643](info): set up 2 sockets
Jun 20 03:06:30 dist-test-slave-xv68 krb5kdc[8643](info): commencing operation
krb5kdc: starting...
W20260620 03:06:31.628335 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:06:31 dist-test-slave-xv68 krb5kdc[8643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:06:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:06:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:06:31.839709 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40535
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:32.152588 8656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:32.152917 8656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:32.153088 8656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:32.160084 8656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:06:32.160214 8656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:32.160322 8656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:32.160423 8656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:06:32.160497 8656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:06:32.170490 8656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:32.171820 8656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:32.173171 8656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:32.184692 8661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:32.184741 8662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:32.184779 8664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:32.185678 8656 server_base.cc:1034] running on GCE node
I20260620 03:06:32.186451 8656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:32.187923 8656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:32.189124 8656 hybrid_clock.cc:648] HybridClock initialized: now 1781924792189083 us; error 42 us; skew 500 ppm
I20260620 03:06:32.189466 8656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:32 dist-test-slave-xv68 krb5kdc[8643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:32.195552 8656 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:06:32.197114 8656 webserver.cc:460] Webserver started at http://127.0.97.62:35057/ using document root <none> and password file <none>
I20260620 03:06:32.197800 8656 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:32.197935 8656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:32.198278 8656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:32.201329 8656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "2aada87120364ebc975e090f56810b67"
format_stamp: "Formatted at 2026-06-20 03:06:32 on dist-test-slave-xv68"
server_key: "8c797984cb30c8a8c0ba7d4b327ad2e8"
server_key_iv: "1f1ce6f16f9cc88f03f5f49a1e7b4ac7"
server_key_version: "encryptionkey@0"
I20260620 03:06:32.202165 8656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "2aada87120364ebc975e090f56810b67"
format_stamp: "Formatted at 2026-06-20 03:06:32 on dist-test-slave-xv68"
server_key: "8c797984cb30c8a8c0ba7d4b327ad2e8"
server_key_iv: "1f1ce6f16f9cc88f03f5f49a1e7b4ac7"
server_key_version: "encryptionkey@0"
I20260620 03:06:32.207430 8656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 03:06:32.210862 8671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:32.212216 8656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 03:06:32.212396 8656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "2aada87120364ebc975e090f56810b67"
format_stamp: "Formatted at 2026-06-20 03:06:32 on dist-test-slave-xv68"
server_key: "8c797984cb30c8a8c0ba7d4b327ad2e8"
server_key_iv: "1f1ce6f16f9cc88f03f5f49a1e7b4ac7"
server_key_version: "encryptionkey@0"
I20260620 03:06:32.212594 8656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:32.249913 8656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:32.253952 8656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:32.254266 8656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:32.274015 8656 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40535
I20260620 03:06:32.274101 8722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40535 every 8 connection(s)
I20260620 03:06:32.275497 8656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:06:32.279426 8723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:32.279781 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 8656
I20260620 03:06:32.279996 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:06:32.280611 388 external_mini_cluster.cc:1428] Setting key a65353aee11ae282ea9057611850f8c2
I20260620 03:06:32.293217 8723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67: Bootstrap starting.
Jun 20 03:06:32 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:06:32.296780 8723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:32.297806 8723 log.cc:824] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:32.300904 8723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67: No bootstrap required, opened a new log
I20260620 03:06:32.304487 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:32.282682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.0.97.62:40535)
0620 03:06:32.283465 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:32.283482 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:32.283520 (+ 38us) server_negotiation.cc:407] Connection header received
0620 03:06:32.285339 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:32.285363 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:32.285747 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:32.286067 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:32.288118 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:32.289374 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:32.291214 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:32.291906 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:32.292040 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:32.296541 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:32.296549 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:32.296564 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:32.296599 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:32.300132 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:32.300761 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:32.300768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:32.300776 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:32.300993 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:32.301373 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:32.301380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:32.301386 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:32.302056 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:32.302336 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:32.302656 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 03:06:32.303104 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":360,"threads_started":1}
I20260620 03:06:32.309912 8723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aada87120364ebc975e090f56810b67" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40535 } }
I20260620 03:06:32.310236 8723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:32.310351 8723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aada87120364ebc975e090f56810b67, State: Initialized, Role: FOLLOWER
I20260620 03:06:32.310864 8723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aada87120364ebc975e090f56810b67" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40535 } }
I20260620 03:06:32.311100 8723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:06:32.311239 8723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:06:32.311419 8723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:32.313700 8723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aada87120364ebc975e090f56810b67" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40535 } }
I20260620 03:06:32.314173 8723 leader_election.cc:302] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2aada87120364ebc975e090f56810b67; no voters:
I20260620 03:06:32.314754 8723 leader_election.cc:288] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:06:32.314890 8728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:32.315585 8728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [term 1 LEADER]: Becoming Leader. State: Replica: 2aada87120364ebc975e090f56810b67, State: Running, Role: LEADER
I20260620 03:06:32.316195 8728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aada87120364ebc975e090f56810b67" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40535 } }
I20260620 03:06:32.316967 8723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:06:32.319020 8729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2aada87120364ebc975e090f56810b67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aada87120364ebc975e090f56810b67" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40535 } } }
I20260620 03:06:32.319548 8729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:32.319031 8730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aada87120364ebc975e090f56810b67. Latest consensus state: current_term: 1 leader_uuid: "2aada87120364ebc975e090f56810b67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aada87120364ebc975e090f56810b67" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40535 } } }
I20260620 03:06:32.319769 8730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:32.320447 8733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:06:32.326038 8733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:06:32.334971 8733 catalog_manager.cc:1347] Generated new cluster ID: 58b479b06f724c9a89cab349003ce7c2
I20260620 03:06:32.335114 8733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:06:32.357386 8733 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:06:32.359244 8733 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:06:32.375103 8733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67: Generated new TSK 0
I20260620 03:06:32.375968 8733 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:06:32.438864 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40535
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:32.753917 8749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:32.754179 8749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:32.754354 8749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:32.762048 8749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:32.762176 8749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:32.762360 8749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:06:32.771670 8749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:32.772921 8749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:32.774176 8749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:32.786644 8754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:32.786597 8755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:32.786661 8757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:32.787354 8749 server_base.cc:1034] running on GCE node
I20260620 03:06:32.788022 8749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:32.788849 8749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:32.790030 8749 hybrid_clock.cc:648] HybridClock initialized: now 1781924792789984 us; error 58 us; skew 500 ppm
I20260620 03:06:32.790342 8749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:32 dist-test-slave-xv68 krb5kdc[8643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:32.793433 8749 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:06:32.794838 8749 webserver.cc:460] Webserver started at http://127.0.97.1:41257/ using document root <none> and password file <none>
I20260620 03:06:32.795567 8749 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:32.795706 8749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:32.796011 8749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:32.798763 8749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "717feead9edb4f86b9e189ce75e3976c"
format_stamp: "Formatted at 2026-06-20 03:06:32 on dist-test-slave-xv68"
server_key: "2b3fddff98cab163ed853a150bb98dd1"
server_key_iv: "40149d5da66dbe6b1d21c8bb1c52f2c4"
server_key_version: "encryptionkey@0"
I20260620 03:06:32.799561 8749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "717feead9edb4f86b9e189ce75e3976c"
format_stamp: "Formatted at 2026-06-20 03:06:32 on dist-test-slave-xv68"
server_key: "2b3fddff98cab163ed853a150bb98dd1"
server_key_iv: "40149d5da66dbe6b1d21c8bb1c52f2c4"
server_key_version: "encryptionkey@0"
I20260620 03:06:32.804572 8749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:06:32.807824 8764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:32.809090 8749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:06:32.809253 8749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "717feead9edb4f86b9e189ce75e3976c"
format_stamp: "Formatted at 2026-06-20 03:06:32 on dist-test-slave-xv68"
server_key: "2b3fddff98cab163ed853a150bb98dd1"
server_key_iv: "40149d5da66dbe6b1d21c8bb1c52f2c4"
server_key_version: "encryptionkey@0"
I20260620 03:06:32.809455 8749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:32.830222 8749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:32.834022 8749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:32.834321 8749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:32.835232 8749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:32.836491 8749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:32.836597 8749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:32.836736 8749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:32.836807 8749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:32.869341 8749 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35601
I20260620 03:06:32.869391 8876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35601 every 8 connection(s)
I20260620 03:06:32.870570 8749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:06:32.878446 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 8749
I20260620 03:06:32.878647 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:06:32.879251 388 external_mini_cluster.cc:1428] Setting key 0115f7d5b2e09b49c7af103f2193a7fb
Jun 20 03:06:32 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:06:32.889490 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:32.872939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59371 (local address 127.0.97.62:40535)
0620 03:06:32.873353 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:32.873357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:32.874427 (+ 1070us) server_negotiation.cc:407] Connection header received
0620 03:06:32.876252 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:32.876257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:32.876345 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:32.876463 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:32.877840 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:32.878597 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:32.880291 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:32.880690 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:32.880822 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:32.884040 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:32.884045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:32.884048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:32.884077 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:32.886563 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:32.887251 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:32.887257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:32.887260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:32.887442 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:32.887861 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:32.887865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:32.887869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:32.888146 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:32.888298 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:32.888945 (+ 647us) server_negotiation.cc:299] Negotiation successful
0620 03:06:32.889156 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":212,"threads_started":1}
I20260620 03:06:32.890358 8880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:32.873526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:59371)
0620 03:06:32.874282 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:32.874330 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:32.875995 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:32.876692 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:32.876699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:32.877155 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:32.877675 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:32.877690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:32.878748 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:32.878760 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:32.880155 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:32.880166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:32.880908 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:32.880916 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:32.881141 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:32.881775 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:32.881799 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:32.883826 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:32.886714 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:32.886722 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:32.886736 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:32.887131 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:32.887553 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:32.887558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:32.887562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:32.887739 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:32.888271 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:32.888279 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:32.888591 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 03:06:32.888840 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 03:06:32.889307 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":271,"threads_started":1}
I20260620 03:06:32.891871 8877 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40535
I20260620 03:06:32.892172 8877 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:32.892992 8877 heartbeater.cc:511] Master 127.0.97.62:40535 requested a full tablet report, sending...
I20260620 03:06:32.895366 8688 ts_manager.cc:194] Registered new tserver with Master: 717feead9edb4f86b9e189ce75e3976c (127.0.97.1:35601)
I20260620 03:06:32.897567 8688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59371
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:06:32.930039 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40535
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:33.249336 8883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:33.249603 8883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:33.249763 8883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:33.257266 8883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:33.257383 8883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:33.257568 8883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:06:33.267246 8883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:33.268550 8883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:33.269809 8883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:33.284390 8889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:33.284945 8891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:33.285135 8888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:33.286072 8883 server_base.cc:1034] running on GCE node
I20260620 03:06:33.286710 8883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:33.287741 8883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:33.288995 8883 hybrid_clock.cc:648] HybridClock initialized: now 1781924793288935 us; error 50 us; skew 500 ppm
I20260620 03:06:33.289527 8883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:33 dist-test-slave-xv68 krb5kdc[8643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:33.293109 8883 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:06:33.294595 8883 webserver.cc:460] Webserver started at http://127.0.97.2:41331/ using document root <none> and password file <none>
I20260620 03:06:33.295276 8883 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:33.295482 8883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:33.295794 8883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:33.298715 8883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "8f152a942cf641d38dbbab902d0a5cb9"
format_stamp: "Formatted at 2026-06-20 03:06:33 on dist-test-slave-xv68"
server_key: "2e20ea3a7e19835e8f3c3bf372152726"
server_key_iv: "bad5f77ba9850a55b3a82b5cdc3c27c1"
server_key_version: "encryptionkey@0"
I20260620 03:06:33.299562 8883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8f152a942cf641d38dbbab902d0a5cb9"
format_stamp: "Formatted at 2026-06-20 03:06:33 on dist-test-slave-xv68"
server_key: "2e20ea3a7e19835e8f3c3bf372152726"
server_key_iv: "bad5f77ba9850a55b3a82b5cdc3c27c1"
server_key_version: "encryptionkey@0"
I20260620 03:06:33.304560 8883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 03:06:33.307953 8898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:33.309316 8883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:06:33.309486 8883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "8f152a942cf641d38dbbab902d0a5cb9"
format_stamp: "Formatted at 2026-06-20 03:06:33 on dist-test-slave-xv68"
server_key: "2e20ea3a7e19835e8f3c3bf372152726"
server_key_iv: "bad5f77ba9850a55b3a82b5cdc3c27c1"
server_key_version: "encryptionkey@0"
I20260620 03:06:33.309679 8883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:33.342772 8883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:33.346707 8883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:33.347013 8883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:33.347990 8883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:33.349278 8883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:33.349382 8883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:33.349525 8883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:33.349601 8883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:33.382068 8883 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41789
I20260620 03:06:33.382105 9010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41789 every 8 connection(s)
I20260620 03:06:33.383416 8883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:06:33.390344 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 8883
I20260620 03:06:33.390527 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:06:33.391047 388 external_mini_cluster.cc:1428] Setting key 040ac0105433a974a51611d9583f0d0c
Jun 20 03:06:33 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:06:33.405473 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:33.385912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41393 (local address 127.0.97.62:40535)
0620 03:06:33.386091 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:33.386095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:33.387238 (+ 1143us) server_negotiation.cc:407] Connection header received
0620 03:06:33.389006 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:33.389010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:33.389077 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:33.389165 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:33.390773 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:33.391793 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:33.393494 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:33.393943 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:33.394127 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:33.398815 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:33.398824 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:33.398841 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:33.398885 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:33.402219 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:33.403141 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:33.403148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:33.403151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:33.403284 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:33.403797 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:33.403801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:33.403803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:33.404080 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:33.404187 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:33.405059 (+ 872us) server_negotiation.cc:299] Negotiation successful
0620 03:06:33.405237 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 03:06:33.407016 9013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:33.386433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:41393)
0620 03:06:33.387088 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:33.387126 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:33.388754 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:33.389405 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:33.389414 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:33.389959 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:33.390555 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:33.390586 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:33.391947 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:33.391953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:33.393340 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:33.393357 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:33.394203 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:33.394212 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:33.394570 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:33.395467 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:33.395517 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:33.398535 (+ 3018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:33.402368 (+ 3833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:33.402379 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:33.402399 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:33.403010 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:33.403434 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:33.403441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:33.403447 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:33.403679 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:33.404221 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:33.404232 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:33.404697 (+ 465us) client_negotiation.cc:769] Sending connection context
0620 03:06:33.405001 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 03:06:33.405569 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":265,"threads_started":1}
I20260620 03:06:33.408545 9011 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40535
I20260620 03:06:33.408857 9011 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:33.409562 9011 heartbeater.cc:511] Master 127.0.97.62:40535 requested a full tablet report, sending...
I20260620 03:06:33.410943 8688 ts_manager.cc:194] Registered new tserver with Master: 8f152a942cf641d38dbbab902d0a5cb9 (127.0.97.2:41789)
I20260620 03:06:33.411818 8688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41393
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:06:33.442817 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40535
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:33.761072 9016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:33.761503 9016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:33.761675 9016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:33.768805 9016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:33.768959 9016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:33.769138 9016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:06:33.778705 9016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:33.780081 9016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:33.781420 9016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:33.794149 9021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:33.794063 9022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:33.794114 9024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:33.795142 9016 server_base.cc:1034] running on GCE node
I20260620 03:06:33.795781 9016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:33.796680 9016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:33.797870 9016 hybrid_clock.cc:648] HybridClock initialized: now 1781924793797808 us; error 53 us; skew 500 ppm
I20260620 03:06:33.798182 9016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:33 dist-test-slave-xv68 krb5kdc[8643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:33.801384 9016 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:06:33.802722 9016 webserver.cc:460] Webserver started at http://127.0.97.3:39703/ using document root <none> and password file <none>
I20260620 03:06:33.803421 9016 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:33.803577 9016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:33.803895 9016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:33.806710 9016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "cb15c49a71a3480291050ef777910bb2"
format_stamp: "Formatted at 2026-06-20 03:06:33 on dist-test-slave-xv68"
server_key: "8927e3281797287621c8d8ff4c2ed4c9"
server_key_iv: "c3dd69db9b3e709f5170911fd56d5ac2"
server_key_version: "encryptionkey@0"
I20260620 03:06:33.807566 9016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cb15c49a71a3480291050ef777910bb2"
format_stamp: "Formatted at 2026-06-20 03:06:33 on dist-test-slave-xv68"
server_key: "8927e3281797287621c8d8ff4c2ed4c9"
server_key_iv: "c3dd69db9b3e709f5170911fd56d5ac2"
server_key_version: "encryptionkey@0"
I20260620 03:06:33.812597 9016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:06:33.816133 9031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:33.817591 9016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:06:33.817737 9016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "cb15c49a71a3480291050ef777910bb2"
format_stamp: "Formatted at 2026-06-20 03:06:33 on dist-test-slave-xv68"
server_key: "8927e3281797287621c8d8ff4c2ed4c9"
server_key_iv: "c3dd69db9b3e709f5170911fd56d5ac2"
server_key_version: "encryptionkey@0"
I20260620 03:06:33.817937 9016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:33.839040 9016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:33.843130 9016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:33.843499 9016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:33.844964 9016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:33.846244 9016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:33.846318 9016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:33.846457 9016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:33.846514 9016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:33.878911 9016 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46279
I20260620 03:06:33.878993 9143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46279 every 8 connection(s)
I20260620 03:06:33.880173 9016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:06:33.882687 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 9016
I20260620 03:06:33.882889 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:06:33.883373 388 external_mini_cluster.cc:1428] Setting key a30dc9023dbd025c0be2f2d56604fee3
Jun 20 03:06:33 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:06:33.899170 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:33.882591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48685 (local address 127.0.97.62:40535)
0620 03:06:33.882808 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:33.882815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:33.884075 (+ 1260us) server_negotiation.cc:407] Connection header received
0620 03:06:33.885949 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:33.885954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:33.886034 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:33.886148 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:33.887565 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:33.888287 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:33.889770 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:33.890174 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:33.890301 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:33.893630 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:33.893635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:33.893639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:33.893670 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:33.896119 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:33.896986 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:33.896992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:33.896997 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:33.897131 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:33.897543 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:33.897554 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:33.897557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:33.897810 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:33.897962 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:33.898611 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 03:06:33.898816 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 03:06:33.900163 9146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:33.883078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.3:48685)
0620 03:06:33.883933 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:33.883983 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:33.885685 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:33.886371 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:33.886378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:33.886840 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:33.887418 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:33.887436 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:33.888441 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:33.888446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:33.889633 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:33.889644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:33.890386 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:33.890395 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:33.890627 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:33.891246 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:33.891271 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:33.893434 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:33.896307 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:33.896316 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:33.896333 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:33.896794 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:33.897242 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:33.897249 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:33.897252 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:33.897430 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:33.897940 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:33.897947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:33.898255 (+ 308us) client_negotiation.cc:769] Sending connection context
0620 03:06:33.898502 (+ 247us) client_negotiation.cc:240] Negotiation successful
0620 03:06:33.899025 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":406,"threads_started":1}
I20260620 03:06:33.901471 8877 heartbeater.cc:503] Master 127.0.97.62:40535 was elected leader, sending a full tablet report...
I20260620 03:06:33.901510 9144 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40535
I20260620 03:06:33.901798 9144 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:33.902586 9144 heartbeater.cc:511] Master 127.0.97.62:40535 requested a full tablet report, sending...
I20260620 03:06:33.903976 8688 ts_manager.cc:194] Registered new tserver with Master: cb15c49a71a3480291050ef777910bb2 (127.0.97.3:46279)
I20260620 03:06:33.904731 8688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48685
I20260620 03:06:33.910377 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 03:06:34.047947 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 8749
Jun 20 03:06:34 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:06:34.063740 9154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:34.049174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37878 (local address 127.0.97.1:35601)
0620 03:06:34.049844 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:34.049851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:34.049879 (+ 28us) server_negotiation.cc:407] Connection header received
0620 03:06:34.049937 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:34.049944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:34.050153 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:34.050285 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:34.051615 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:34.052621 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:34.053791 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:34.054528 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:34.054633 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:34.058484 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:34.058495 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:34.058510 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:34.058548 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:34.061333 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:34.061957 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:34.061964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:34.061966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:34.062084 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:34.062437 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:34.062440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:34.062442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:34.062740 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:34.062982 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:34.063204 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 03:06:34.063468 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":359,"threads_started":1}
I20260620 03:06:34.064992 8872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:34.146416 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 8749
I20260620 03:06:34.163290 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 8883
Jun 20 03:06:34 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:06:34.178980 9156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:34.164273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41698 (local address 127.0.97.2:41789)
0620 03:06:34.164783 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:34.164790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:34.164813 (+ 23us) server_negotiation.cc:407] Connection header received
0620 03:06:34.164905 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:34.164911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:34.165102 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:34.165226 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:34.166405 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:34.167476 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:34.168694 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:34.169242 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:34.169337 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:34.173374 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:34.173385 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:34.173398 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:34.173427 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:34.176328 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:34.176971 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:34.176978 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:34.176981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:34.177107 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:34.177420 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:34.177423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:34.177425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:34.177746 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:34.177987 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:34.178338 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 03:06:34.178592 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":264,"threads_started":1}
I20260620 03:06:34.180323 9006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:34.263609 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 8883
I20260620 03:06:34.279907 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 9016
Jun 20 03:06:34 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:06:34.296298 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:34.280974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53732 (local address 127.0.97.3:46279)
0620 03:06:34.281614 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:34.281621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:34.281648 (+ 27us) server_negotiation.cc:407] Connection header received
0620 03:06:34.281703 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:34.281710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:34.281924 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:34.282049 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:34.283329 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:34.284358 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:34.285605 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:34.286155 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:34.286250 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:34.290529 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:34.290540 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:34.290555 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:34.290591 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:34.293446 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:34.294142 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:34.294149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:34.294152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:34.294277 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:34.294648 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:34.294654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:34.294658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:34.294998 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:34.295328 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:34.295687 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 03:06:34.295962 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":353,"threads_started":1}
I20260620 03:06:34.297585 9139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:34.379233 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 9016
I20260620 03:06:34.395481 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 8656
I20260620 03:06:34.396687 8718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:34.498111 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 8656
I20260620 03:06:34.518712 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40535
--webserver_interface=127.0.97.62
--webserver_port=35057
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:34.829778 9161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:34.830125 9161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:34.830296 9161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:34.837572 9161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:06:34.837714 9161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:34.837823 9161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:34.837920 9161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:06:34.837993 9161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:06:34.847879 9161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:34.849218 9161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:34.850541 9161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:34.862054 9167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:34.862124 9166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:34.862080 9169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:34.862818 9161 server_base.cc:1034] running on GCE node
I20260620 03:06:34.863613 9161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:34.864987 9161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:34.866179 9161 hybrid_clock.cc:648] HybridClock initialized: now 1781924794866141 us; error 33 us; skew 500 ppm
I20260620 03:06:34.866521 9161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:34 dist-test-slave-xv68 krb5kdc[8643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:34.869669 9161 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:06:34.871070 9161 webserver.cc:460] Webserver started at http://127.0.97.62:35057/ using document root <none> and password file <none>
I20260620 03:06:34.871764 9161 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:34.871904 9161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:34.877148 9161 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 03:06:34.880015 9176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:34.881419 9161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:06:34.881593 9161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "2aada87120364ebc975e090f56810b67"
format_stamp: "Formatted at 2026-06-20 03:06:32 on dist-test-slave-xv68"
server_key: "8c797984cb30c8a8c0ba7d4b327ad2e8"
server_key_iv: "1f1ce6f16f9cc88f03f5f49a1e7b4ac7"
server_key_version: "encryptionkey@0"
I20260620 03:06:34.882519 9161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:34.899837 9161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:34.903779 9161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:34.904098 9161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:34.922035 9161 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40535
I20260620 03:06:34.922068 9227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40535 every 8 connection(s)
I20260620 03:06:34.923547 9161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:06:34.928419 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 9161
I20260620 03:06:34.929329 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35601
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40535
--webserver_port=41257
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:06:34.930544 9228 sys_catalog.cc:263] Verifying existing consensus state
I20260620 03:06:34.933919 9228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67: Bootstrap starting.
I20260620 03:06:34.952214 9228 log.cc:824] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:34.960722 9228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 03:06:34.961244 9228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67: Bootstrap complete.
I20260620 03:06:34.969935 9228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aada87120364ebc975e090f56810b67" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40535 } }
I20260620 03:06:34.970701 9228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aada87120364ebc975e090f56810b67, State: Initialized, Role: FOLLOWER
I20260620 03:06:34.971385 9228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aada87120364ebc975e090f56810b67" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40535 } }
I20260620 03:06:34.971608 9228 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:06:34.971714 9228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:06:34.971899 9228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [term 1 FOLLOWER]: Advancing to term 2
I20260620 03:06:34.975580 9228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aada87120364ebc975e090f56810b67" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40535 } }
I20260620 03:06:34.976046 9228 leader_election.cc:302] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2aada87120364ebc975e090f56810b67; no voters:
I20260620 03:06:34.976656 9228 leader_election.cc:288] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 03:06:34.976778 9232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 03:06:34.978128 9232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [term 2 LEADER]: Becoming Leader. State: Replica: 2aada87120364ebc975e090f56810b67, State: Running, Role: LEADER
I20260620 03:06:34.978727 9232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aada87120364ebc975e090f56810b67" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40535 } }
I20260620 03:06:34.978858 9228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:06:34.981320 9234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aada87120364ebc975e090f56810b67. Latest consensus state: current_term: 2 leader_uuid: "2aada87120364ebc975e090f56810b67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aada87120364ebc975e090f56810b67" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40535 } } }
I20260620 03:06:34.982244 9234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:34.981643 9233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2aada87120364ebc975e090f56810b67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aada87120364ebc975e090f56810b67" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40535 } } }
I20260620 03:06:34.982805 9233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:34.987811 9238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:06:34.994460 9238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:06:34.995882 9238 catalog_manager.cc:1259] Loaded cluster ID: 58b479b06f724c9a89cab349003ce7c2
I20260620 03:06:34.996040 9238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:06:34.999910 9238 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:06:35.001345 9238 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2aada87120364ebc975e090f56810b67: Loaded TSK: 0
I20260620 03:06:35.002830 9238 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 03:06:35.279502 9230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:35.279814 9230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:35.280014 9230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:35.287477 9230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:35.287575 9230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:35.287741 9230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:06:35.297495 9230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41257
--tserver_master_addrs=127.0.97.62:40535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:35.298736 9230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:35.300078 9230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:35.315150 9254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:35.315127 9255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:35.320190 9257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:35.320947 9230 server_base.cc:1034] running on GCE node
I20260620 03:06:35.321607 9230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:35.322654 9230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:35.323933 9230 hybrid_clock.cc:648] HybridClock initialized: now 1781924795323879 us; error 44 us; skew 500 ppm
I20260620 03:06:35.324275 9230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:35 dist-test-slave-xv68 krb5kdc[8643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924795, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:35.327553 9230 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 03:06:35.329203 9230 webserver.cc:460] Webserver started at http://127.0.97.1:41257/ using document root <none> and password file <none>
I20260620 03:06:35.329983 9230 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:35.330127 9230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:35.335716 9230 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 03:06:35.338569 9264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:35.340097 9230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:06:35.340253 9230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "717feead9edb4f86b9e189ce75e3976c"
format_stamp: "Formatted at 2026-06-20 03:06:32 on dist-test-slave-xv68"
server_key: "2b3fddff98cab163ed853a150bb98dd1"
server_key_iv: "40149d5da66dbe6b1d21c8bb1c52f2c4"
server_key_version: "encryptionkey@0"
I20260620 03:06:35.341210 9230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:35.366431 9230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:35.370352 9230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:35.370623 9230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:35.371703 9230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:35.373044 9230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:35.373131 9230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:35.373274 9230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:35.373378 9230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:35.414844 9230 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35601
I20260620 03:06:35.414940 9376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35601 every 8 connection(s)
I20260620 03:06:35.416349 9230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:06:35.421569 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 9230
I20260620 03:06:35.422194 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41789
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40535
--webserver_port=41331
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:06:35 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:35 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:35.435175 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:35.419491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:54929)
0620 03:06:35.420307 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:35.420365 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:35.422124 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:35.423629 (+ 1505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:35.423641 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:35.424221 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:35.424765 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:35.424782 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.426361 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.426365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:35.427923 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:35.427936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.429015 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.429032 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:35.429294 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:35.429970 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:35.429993 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:35.433829 (+ 3836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":336,"threads_started":1}
W20260620 03:06:35.435197 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:35.419412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54929 (local address 127.0.97.62:40535)
0620 03:06:35.420256 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:35.420275 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:35.420493 (+ 218us) server_negotiation.cc:407] Connection header received
0620 03:06:35.422755 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:35.422768 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:35.423112 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:35.423492 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:35.424965 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.426191 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.428102 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.428775 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.428982 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:35.434225 (+ 5243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54929: BlockingRecv error: recv got EOF from 127.0.97.1:54929 (error 108)
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":301,"threads_started":1}
W20260620 03:06:35.436110 9377 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40535 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:40535: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 03:06:35 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:35 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:35.444553 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:35.436744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:39147)
0620 03:06:35.436917 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:35.436942 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:35.437053 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:35.437423 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:35.437427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:35.437769 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:35.437912 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:35.437920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.438930 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.438935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:35.439909 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:35.439921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.440595 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.440600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:35.440768 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:35.441339 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:35.441357 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:35.444236 (+ 2879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 03:06:35.444628 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:35.436966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39147 (local address 127.0.97.62:40535)
0620 03:06:35.437159 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:35.437182 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:35.437197 (+ 15us) server_negotiation.cc:407] Connection header received
0620 03:06:35.437241 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:35.437244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:35.437298 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:35.437408 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:35.438048 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.438783 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.440044 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.440448 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.440585 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:35.444424 (+ 3839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39147: BlockingRecv error: recv got EOF from 127.0.97.1:39147 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 03:06:35 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:35 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:35.452889 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:35.445359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:49415)
0620 03:06:35.445539 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:35.445556 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:35.445643 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:35.445969 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:35.445973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:35.446277 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:35.446414 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:35.446422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.447378 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.447387 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:35.448368 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:35.448379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.449078 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.449083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:35.449250 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:35.449846 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:35.449862 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:35.452578 (+ 2716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 03:06:35.453076 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:35.445417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49415 (local address 127.0.97.62:40535)
0620 03:06:35.445572 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:35.445578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:35.445596 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:06:35.445759 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:35.445765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:35.445844 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:35.445949 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:35.446544 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.447226 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.448503 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.448931 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.449056 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:35.452778 (+ 3722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49415: BlockingRecv error: recv got EOF from 127.0.97.1:49415 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 03:06:35.453199 9377 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 03:06:35.745174 9381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:35.745496 9381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:35.745595 9381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:35.752782 9381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:35.752898 9381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:35.753014 9381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:06:35.762431 9381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41331
--tserver_master_addrs=127.0.97.62:40535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:35.763689 9381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:35.764925 9381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:35.777396 9386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:35.777480 9389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:35.777480 9387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:35.778447 9381 server_base.cc:1034] running on GCE node
I20260620 03:06:35.779006 9381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:35.779953 9381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:35.781178 9381 hybrid_clock.cc:648] HybridClock initialized: now 1781924795781126 us; error 70 us; skew 500 ppm
I20260620 03:06:35.781483 9381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:35 dist-test-slave-xv68 krb5kdc[8643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924795, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:35.784549 9381 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 03:06:35.786049 9381 webserver.cc:460] Webserver started at http://127.0.97.2:41331/ using document root <none> and password file <none>
I20260620 03:06:35.786727 9381 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:35.786871 9381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:35.791868 9381 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:06:35.794377 9396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:35.795662 9381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:06:35.795830 9381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "8f152a942cf641d38dbbab902d0a5cb9"
format_stamp: "Formatted at 2026-06-20 03:06:33 on dist-test-slave-xv68"
server_key: "2e20ea3a7e19835e8f3c3bf372152726"
server_key_iv: "bad5f77ba9850a55b3a82b5cdc3c27c1"
server_key_version: "encryptionkey@0"
I20260620 03:06:35.796720 9381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:35.816181 9381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:35.819957 9381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:35.820221 9381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:35.821161 9381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:35.822407 9381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:35.822528 9381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:35.822662 9381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:35.822750 9381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:35.855440 9381 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41789
I20260620 03:06:35.855521 9508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41789 every 8 connection(s)
I20260620 03:06:35.856844 9381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:06:35.863422 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 9381
I20260620 03:06:35.864006 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46279
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40535
--webserver_port=39703
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:06:35 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:35 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 03:06:35.871893 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:35.859393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52247 (local address 127.0.97.62:40535)
0620 03:06:35.859573 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:35.859579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:35.860751 (+ 1172us) server_negotiation.cc:407] Connection header received
0620 03:06:35.862539 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:35.862543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:35.862607 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:35.862694 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:35.864261 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.864946 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.866442 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.866864 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.866998 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:35.871650 (+ 4652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52247: BlockingRecv error: recv got EOF from 127.0.97.2:52247 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 03:06:35.872605 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:35.859860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:52247)
0620 03:06:35.860588 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:35.860624 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:35.862283 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:35.862894 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:35.862902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:35.863461 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:35.864038 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:35.864055 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.865117 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.865120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:35.866289 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:35.866299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.867104 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.867111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:35.867389 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:35.868059 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:35.868083 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:35.871480 (+ 3397us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":285,"threads_started":1}
W20260620 03:06:35.873320 9509 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40535 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:40535: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 03:06:35 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:35 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 03:06:35.883164 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:35.874035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47045 (local address 127.0.97.62:40535)
0620 03:06:35.874162 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:35.874166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:35.874180 (+ 14us) server_negotiation.cc:407] Connection header received
0620 03:06:35.874279 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:35.874282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:35.874332 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:35.874402 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:35.875243 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.875955 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.877928 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.878302 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.878396 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:35.883000 (+ 4604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47045: BlockingRecv error: recv got EOF from 127.0.97.2:47045 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 03:06:35.883209 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:35.873857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:47045)
0620 03:06:35.874063 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:35.874083 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:35.874180 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:35.874464 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:35.874470 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:35.874903 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:35.875110 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:35.875123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.876097 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.876105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:35.877789 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:35.877806 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.878453 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.878461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:35.878660 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:35.879279 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:35.879323 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:35.882845 (+ 3522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
Jun 20 03:06:35 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:35 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:35.892379 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:35.883902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:47413)
0620 03:06:35.884068 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:35.884087 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:35.884178 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:35.884464 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:35.884468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:35.884757 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:35.884882 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:35.884889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.885885 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.885893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:35.887495 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:35.887508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.888175 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.888181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:35.888376 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:35.889114 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:35.889133 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:35.892044 (+ 2911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 03:06:35.892475 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:35.884005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47413 (local address 127.0.97.62:40535)
0620 03:06:35.884172 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:35.884177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:35.884193 (+ 16us) server_negotiation.cc:407] Connection header received
0620 03:06:35.884285 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:35.884290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:35.884350 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:35.884443 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:35.885016 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.885723 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.887642 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:35.888047 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:35.888135 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:35.892245 (+ 4110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47413: BlockingRecv error: recv got EOF from 127.0.97.2:47413 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 03:06:35.892755 9509 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 03:06:36.192718 9512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:36.193056 9512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:36.193230 9512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:36.200450 9512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:36.200591 9512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:36.200789 9512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:06:36.210176 9512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39703
--tserver_master_addrs=127.0.97.62:40535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:36.211426 9512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:36.212673 9512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:36.225462 9517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:36.225401 9520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:36.225520 9518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:36.226215 9512 server_base.cc:1034] running on GCE node
I20260620 03:06:36.226879 9512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:36.227934 9512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:36.229147 9512 hybrid_clock.cc:648] HybridClock initialized: now 1781924796229101 us; error 49 us; skew 500 ppm
I20260620 03:06:36.229481 9512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:36 dist-test-slave-xv68 krb5kdc[8643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924796, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:36.232618 9512 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 03:06:36.234050 9512 webserver.cc:460] Webserver started at http://127.0.97.3:39703/ using document root <none> and password file <none>
I20260620 03:06:36.234730 9512 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:36.234884 9512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:36.240144 9512 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:06:36.242664 9527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:36.244029 9512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:06:36.244205 9512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "cb15c49a71a3480291050ef777910bb2"
format_stamp: "Formatted at 2026-06-20 03:06:33 on dist-test-slave-xv68"
server_key: "8927e3281797287621c8d8ff4c2ed4c9"
server_key_iv: "c3dd69db9b3e709f5170911fd56d5ac2"
server_key_version: "encryptionkey@0"
I20260620 03:06:36.245177 9512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:36.275162 9512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:36.279803 9512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:36.280114 9512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:36.281136 9512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:36.282531 9512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:36.282663 9512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:36.282802 9512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:36.282897 9512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:36.319593 9512 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46279
I20260620 03:06:36.319650 9639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46279 every 8 connection(s)
I20260620 03:06:36.321121 9512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:06:36.326773 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 9512
Jun 20 03:06:36 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:36 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 03:06:36.338140 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:36.323673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57961 (local address 127.0.97.62:40535)
0620 03:06:36.323884 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:36.323891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:36.325022 (+ 1131us) server_negotiation.cc:407] Connection header received
0620 03:06:36.327009 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:36.327047 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:36.327187 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:36.327463 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:36.329483 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.330258 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.332051 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.332534 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.332697 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:36.337813 (+ 5116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57961: BlockingRecv error: recv got EOF from 127.0.97.3:57961 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 03:06:36.338845 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:36.324158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.3:57961)
0620 03:06:36.324864 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:36.324908 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:36.326640 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:36.327911 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:36.327924 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:36.328712 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:36.329295 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:36.329312 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.330442 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.330448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:36.331882 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:36.331894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.332777 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.332786 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:36.333061 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:36.333700 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:36.333724 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:36.337620 (+ 3896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":269,"threads_started":1}
W20260620 03:06:36.339629 9640 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40535 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:40535: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260620 03:06:36.342892 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:36.327879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.0.97.62:40535)
0620 03:06:36.328640 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:36.328645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:36.328663 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:06:36.328719 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:36.328722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:36.328770 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:36.328881 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:36.330301 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.331103 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.332362 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.332808 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.332952 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:36.335872 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:36.335885 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:36.335911 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:36.335962 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:36.339185 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:36.340106 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:36.340117 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:36.340128 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:36.340333 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:36.341022 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:36.341027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:36.341030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:36.341417 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:36.341763 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:36.342159 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 03:06:36.342503 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":527,"threads_started":1}
Jun 20 03:06:36 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:36 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:36.348651 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:36.340271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.3:33619)
0620 03:06:36.340439 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:36.340461 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:36.340563 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:36.340868 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:36.340873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:36.341235 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:36.341377 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:36.341385 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.342746 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.342750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:36.343811 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:36.343822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.344561 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.344566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:36.344735 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:36.345445 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:36.345469 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:36.348360 (+ 2891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 03:06:36.348773 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:36.340376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33619 (local address 127.0.97.62:40535)
0620 03:06:36.340549 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:36.340554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:36.340572 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:06:36.340684 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:36.340688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:36.340737 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:36.340804 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:36.341510 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.342602 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.343947 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.344410 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.344552 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:36.348556 (+ 4004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33619: BlockingRecv error: recv got EOF from 127.0.97.3:33619 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 03:06:36 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:36 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:36.356096 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:36.349361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.3:55167)
0620 03:06:36.349527 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:36.349547 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:36.349634 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:36.349880 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:36.349883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:36.350207 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:36.350344 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:36.350351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.351262 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.351265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:36.352192 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:36.352202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.352790 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.352794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:36.352932 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:36.353374 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:36.353388 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:36.355807 (+ 2419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 03:06:36.356217 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:36.349466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55167 (local address 127.0.97.62:40535)
0620 03:06:36.349665 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:36.349669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:36.349682 (+ 13us) server_negotiation.cc:407] Connection header received
0620 03:06:36.349716 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:36.349719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:36.349765 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:36.349862 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:36.350464 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.351157 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.352303 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.352679 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.352764 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:36.355966 (+ 3202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55167: BlockingRecv error: recv got EOF from 127.0.97.3:55167 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
W20260620 03:06:36.356451 9640 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 03:06:36 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:36 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:36.461916 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:36.453731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:44507)
0620 03:06:36.454246 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:36.454262 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:36.454371 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:36.454654 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:36.454657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:36.455087 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:36.455266 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:36.455273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.456269 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.456273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:36.457225 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:36.457236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.457951 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.457956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:36.458068 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:36.458644 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:36.458660 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:36.461705 (+ 3045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":294,"threads_started":1}
W20260620 03:06:36.462122 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:36.453791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44507 (local address 127.0.97.62:40535)
0620 03:06:36.453992 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:36.453997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:36.454351 (+ 354us) server_negotiation.cc:407] Connection header received
0620 03:06:36.454471 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:36.454474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:36.454536 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:36.454613 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:36.455456 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.456155 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.457356 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.457783 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.457908 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:36.461890 (+ 3982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44507: BlockingRecv error: recv got EOF from 127.0.97.1:44507 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 03:06:36 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:36 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:36.901660 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:36.893296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:45235)
0620 03:06:36.893790 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:36.893806 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:36.893911 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:36.894233 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:36.894236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:36.894681 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:36.894874 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:36.894883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.895911 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.895915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:36.896935 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:36.896947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.897635 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.897640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:36.897756 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:36.898414 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:36.898431 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:36.901368 (+ 2937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":279,"threads_started":1}
W20260620 03:06:36.901986 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:36.893380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45235 (local address 127.0.97.62:40535)
0620 03:06:36.893579 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:36.893583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:36.893875 (+ 292us) server_negotiation.cc:407] Connection header received
0620 03:06:36.894046 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:36.894050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:36.894111 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:36.894194 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:36.895047 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.895771 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.897076 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:36.897476 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:36.897565 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:36.901613 (+ 4048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45235: BlockingRecv error: recv got EOF from 127.0.97.2:45235 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 03:06:37 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:37 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:37.370716 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:37.357099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.3:46541)
0620 03:06:37.357760 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:37.357777 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:37.357930 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:37.358273 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:37.358277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:37.358755 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:37.358936 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:37.358944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:37.360005 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:37.360013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:37.361018 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:37.361029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:37.361826 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:37.361832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:37.361995 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:37.366754 (+ 4759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:37.366774 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:37.370368 (+ 3594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":418,"threads_started":1}
W20260620 03:06:37.370793 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:37.357121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46541 (local address 127.0.97.62:40535)
0620 03:06:37.357329 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:37.357338 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:37.357867 (+ 529us) server_negotiation.cc:407] Connection header received
0620 03:06:37.358061 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:37.358066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:37.358134 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:37.358237 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:37.359077 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:37.359855 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:37.361159 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:37.361630 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:37.361756 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:37.370558 (+ 8802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46541: BlockingRecv error: recv got EOF from 127.0.97.3:46541 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 03:06:37 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:37 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:37.470830 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:37.462816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:33007)
0620 03:06:37.463318 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:37.463334 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:37.463444 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:37.463823 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:37.463828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:37.464311 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:37.464489 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:37.464499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:37.465678 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:37.465681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:37.466579 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:37.466590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:37.467249 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:37.467253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:37.467380 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:37.467900 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:37.467916 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:37.470577 (+ 2661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":278,"threads_started":1}
W20260620 03:06:37.470973 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:37.462904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33007 (local address 127.0.97.62:40535)
0620 03:06:37.463169 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:37.463175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:37.463402 (+ 227us) server_negotiation.cc:407] Connection header received
0620 03:06:37.463586 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:37.463590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:37.463657 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:37.463735 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:37.464606 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:37.465558 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:37.466698 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:37.467121 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:37.467219 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:37.470739 (+ 3520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33007: BlockingRecv error: recv got EOF from 127.0.97.1:33007 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 03:06:37 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:37 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:37.912457 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:37.902594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:50283)
0620 03:06:37.903261 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:37.903277 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:37.903396 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:37.903672 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:37.903676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:37.904081 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:37.904221 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:37.904229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:37.905295 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:37.905300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:37.906230 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:37.906241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:37.907191 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:37.907197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:37.907382 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:37.908083 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:37.908105 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:37.912100 (+ 3995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":444,"threads_started":1}
W20260620 03:06:37.912585 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:37.902665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50283 (local address 127.0.97.62:40535)
0620 03:06:37.902838 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:37.902844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:37.903366 (+ 522us) server_negotiation.cc:407] Connection header received
0620 03:06:37.903492 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:37.903495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:37.903554 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:37.903632 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:37.904373 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:37.905135 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:37.906374 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:37.907031 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:37.907152 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:37.912348 (+ 5196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50283: BlockingRecv error: recv got EOF from 127.0.97.2:50283 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 03:06:38 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:38 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:38.380074 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:38.371734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.3:44433)
0620 03:06:38.372263 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:38.372280 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:38.372393 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:38.372710 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:38.372713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:38.373118 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:38.373261 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:38.373268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:38.374307 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:38.374311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:38.375186 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:38.375196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:38.375874 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:38.375878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:38.375996 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:38.376629 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:38.376646 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:38.379738 (+ 3092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":296,"threads_started":1}
W20260620 03:06:38.380193 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:38.371916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44433 (local address 127.0.97.62:40535)
0620 03:06:38.372173 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:38.372179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:38.372358 (+ 179us) server_negotiation.cc:407] Connection header received
0620 03:06:38.372498 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:38.372501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:38.372571 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:38.372659 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:38.373393 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:38.374194 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:38.375330 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:38.375746 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:38.375837 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:38.379911 (+ 4074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44433: BlockingRecv error: recv got EOF from 127.0.97.3:44433 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Jun 20 03:06:38 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:38 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:38.480299 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:38.471862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:52797)
0620 03:06:38.472355 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:38.472370 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:38.472536 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:38.472882 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:38.472887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:38.473268 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:38.473428 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:38.473437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:38.474459 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:38.474464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:38.475481 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:38.475493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:38.476333 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:38.476337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:38.476473 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:38.477209 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:38.477226 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:38.480067 (+ 2841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":260,"threads_started":1}
W20260620 03:06:38.480511 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:38.471942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52797 (local address 127.0.97.62:40535)
0620 03:06:38.472195 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:38.472201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:38.472463 (+ 262us) server_negotiation.cc:407] Connection header received
0620 03:06:38.472665 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:38.472670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:38.472739 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:38.472840 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:38.473573 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:38.474291 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:38.475591 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:38.476185 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:38.476286 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:38.480256 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52797: BlockingRecv error: recv got EOF from 127.0.97.1:52797 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 03:06:38 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:38 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:38.921643 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:38.913497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:56823)
0620 03:06:38.914167 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:38.914183 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:38.914304 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:38.914574 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:38.914577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:38.914884 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:38.915062 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:38.915070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:38.916039 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:38.916043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:38.916918 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:38.916931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:38.917581 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:38.917586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:38.917725 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:38.918389 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:38.918406 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:38.921394 (+ 2988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":436,"threads_started":1}
W20260620 03:06:38.921813 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:38.913581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56823 (local address 127.0.97.62:40535)
0620 03:06:38.913803 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:38.913811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:38.914264 (+ 453us) server_negotiation.cc:407] Connection header received
0620 03:06:38.914391 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:38.914394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:38.914453 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:38.914527 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:38.915186 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:38.915925 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:38.917049 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:38.917445 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:38.917535 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:38.921569 (+ 4034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56823: BlockingRecv error: recv got EOF from 127.0.97.2:56823 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 03:06:39 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:39 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:39.389294 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:39.381152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.3:42275)
0620 03:06:39.381641 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:39.381656 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:39.381808 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:39.382164 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:39.382169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:39.382518 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:39.382684 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:39.382692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:39.383734 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:39.383742 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:39.384638 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:39.384649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:39.385384 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:39.385391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:39.385565 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:39.386229 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:39.386247 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:39.388914 (+ 2667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":257,"threads_started":1}
W20260620 03:06:39.389370 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:39.381223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42275 (local address 127.0.97.62:40535)
0620 03:06:39.381482 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:39.381488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:39.381741 (+ 253us) server_negotiation.cc:407] Connection header received
0620 03:06:39.381930 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:39.381936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:39.382004 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:39.382107 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:39.382825 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:39.383613 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:39.384774 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:39.385238 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:39.385360 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:39.389142 (+ 3782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42275: BlockingRecv error: recv got EOF from 127.0.97.3:42275 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Jun 20 03:06:39 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:39 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:39.489508 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:39.481194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:59279)
0620 03:06:39.481657 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:39.481673 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:39.481785 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:39.482117 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:39.482120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:39.482465 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:39.482624 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:39.482631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:39.483684 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:39.483688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:39.484632 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:39.484643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:39.485259 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:39.485264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:39.485461 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:39.486025 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:39.486041 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:39.489223 (+ 3182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":266,"threads_started":1}
W20260620 03:06:39.489682 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:39.481283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59279 (local address 127.0.97.62:40535)
0620 03:06:39.481533 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:39.481540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:39.481773 (+ 233us) server_negotiation.cc:407] Connection header received
0620 03:06:39.481915 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:39.481919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:39.481991 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:39.482067 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:39.482761 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:39.483567 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:39.484749 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:39.485142 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:39.485234 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:39.489462 (+ 4228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59279: BlockingRecv error: recv got EOF from 127.0.97.1:59279 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Jun 20 03:06:39 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:39 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:39.931097 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:39.922668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:44603)
0620 03:06:39.923147 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:39.923163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:39.923337 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:39.923663 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:39.923668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:39.924051 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:39.924204 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:39.924212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:39.925221 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:39.925228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:39.926192 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:39.926202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:39.926954 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:39.926959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:39.927145 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:39.927836 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:39.927853 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:39.930740 (+ 2887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":269,"threads_started":1}
W20260620 03:06:39.931150 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:39.922728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44603 (local address 127.0.97.62:40535)
0620 03:06:39.922923 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:39.922931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:39.923252 (+ 321us) server_negotiation.cc:407] Connection header received
0620 03:06:39.923453 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:39.923457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:39.923532 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:39.923635 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:39.924350 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:39.925083 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:39.926336 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:39.926796 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:39.926946 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:39.930936 (+ 3990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44603: BlockingRecv error: recv got EOF from 127.0.97.2:44603 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 03:06:40 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:40 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:40.398329 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:40.390258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.3:60757)
0620 03:06:40.390722 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:40.390738 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:40.390850 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:40.391116 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:40.391119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:40.391474 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:40.391625 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:40.391632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:40.392659 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:40.392663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:40.393656 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:40.393667 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:40.394417 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:40.394421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:40.394543 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:40.395116 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:40.395133 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:40.398111 (+ 2978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":233,"threads_started":1}
W20260620 03:06:40.398600 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:40.390341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60757 (local address 127.0.97.62:40535)
0620 03:06:40.390567 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:40.390573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:40.390808 (+ 235us) server_negotiation.cc:407] Connection header received
0620 03:06:40.390928 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:40.390932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:40.390998 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:40.391072 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:40.391736 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:40.392530 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:40.393776 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:40.394292 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:40.394382 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:40.398309 (+ 3927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60757: BlockingRecv error: recv got EOF from 127.0.97.3:60757 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 03:06:40 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:40 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:40.499002 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:40.490556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:58785)
0620 03:06:40.491044 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:40.491060 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:40.491174 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:40.491653 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:40.491657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:40.492090 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:40.492256 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:40.492264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:40.493249 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:40.493253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:40.494155 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:40.494167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:40.494815 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:40.494819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:40.494943 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:40.495621 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:40.495637 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:40.498773 (+ 3136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":241,"threads_started":1}
W20260620 03:06:40.499202 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:40.490651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58785 (local address 127.0.97.62:40535)
0620 03:06:40.490890 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:40.490896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:40.491158 (+ 262us) server_negotiation.cc:407] Connection header received
0620 03:06:40.491278 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:40.491282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:40.491441 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:40.491598 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:40.492389 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:40.493116 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:40.494284 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:40.494693 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:40.494784 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:40.498972 (+ 4188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58785: BlockingRecv error: recv got EOF from 127.0.97.1:58785 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Jun 20 03:06:40 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:40 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:40.939858 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:40.932117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:50791)
0620 03:06:40.932642 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:40.932660 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:40.932774 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:40.933066 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:40.933069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:40.933448 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:40.933600 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:40.933608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:40.934549 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:40.934552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:40.935462 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:40.935473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:40.936132 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:40.936138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:40.936313 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:40.936938 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:40.936954 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:40.939631 (+ 2677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":258,"threads_started":1}
W20260620 03:06:40.940024 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:40.932340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50791 (local address 127.0.97.62:40535)
0620 03:06:40.932555 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:40.932563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:40.932739 (+ 176us) server_negotiation.cc:407] Connection header received
0620 03:06:40.932872 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:40.932875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:40.932943 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:40.933035 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:40.933716 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:40.934439 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:40.935578 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:40.935970 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:40.936116 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:40.939802 (+ 3686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50791: BlockingRecv error: recv got EOF from 127.0.97.2:50791 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 03:06:41 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:41 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:41.407652 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:41.399328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.3:52901)
0620 03:06:41.399933 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:41.399964 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:41.400097 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:41.400527 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:41.400534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:41.400933 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:41.401100 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:41.401109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:41.402101 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:41.402106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:41.403122 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:41.403139 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:41.404028 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:41.404034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:41.404181 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:41.404765 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:41.404782 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:41.407417 (+ 2635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":286,"threads_started":1}
W20260620 03:06:41.407779 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:41.399347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52901 (local address 127.0.97.62:40535)
0620 03:06:41.399523 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:41.399531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:41.400045 (+ 514us) server_negotiation.cc:407] Connection header received
0620 03:06:41.400219 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:41.400226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:41.400341 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:41.400432 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:41.401228 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:41.401955 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:41.403258 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:41.403837 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:41.403926 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:41.407574 (+ 3648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52901: BlockingRecv error: recv got EOF from 127.0.97.3:52901 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 03:06:41 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:41 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:41.508131 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:41.499860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:37365)
0620 03:06:41.500312 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:41.500328 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:41.500440 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:41.500787 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:41.500791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:41.501164 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:41.501320 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:41.501328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:41.502331 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:41.502338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:41.503665 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:41.503676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:41.504363 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:41.504368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:41.504501 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:41.505151 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:41.505168 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:41.507883 (+ 2715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":260,"threads_started":1}
W20260620 03:06:41.508301 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:41.499955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37365 (local address 127.0.97.62:40535)
0620 03:06:41.500190 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:41.500196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:41.500399 (+ 203us) server_negotiation.cc:407] Connection header received
0620 03:06:41.500565 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:41.500570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:41.500667 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:41.500744 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:41.501436 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:41.502188 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:41.503789 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:41.504222 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:41.504320 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:41.508066 (+ 3746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37365: BlockingRecv error: recv got EOF from 127.0.97.1:37365 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 03:06:41 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:41 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:41.949169 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:41.940767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:37305)
0620 03:06:41.941545 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:41.941566 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:41.941681 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:41.942034 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:41.942037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:41.942392 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:41.942545 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:41.942553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:41.943588 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:41.943591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:41.944508 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:41.944520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:41.945221 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:41.945225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:41.945355 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:41.945901 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:41.945917 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:41.948796 (+ 2879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":552,"threads_started":1}
W20260620 03:06:41.949312 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:41.940857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37305 (local address 127.0.97.62:40535)
0620 03:06:41.941124 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:41.941134 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:41.941644 (+ 510us) server_negotiation.cc:407] Connection header received
0620 03:06:41.941813 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:41.941817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:41.941878 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:41.941972 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:41.942689 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:41.943472 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:41.944623 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:41.945049 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:41.945143 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:41.949042 (+ 3899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37305: BlockingRecv error: recv got EOF from 127.0.97.2:37305 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 03:06:42 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:42 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:42.421087 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:42.408635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.3:49799)
0620 03:06:42.409208 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:42.409224 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:42.409342 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:42.409629 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:42.409632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:42.409975 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:42.410164 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:42.410182 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:42.411225 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:42.411229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:42.412175 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:42.412186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:42.412913 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:42.412917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:42.413063 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:42.418035 (+ 4972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:42.418052 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:42.420877 (+ 2825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":346,"threads_started":1}
W20260620 03:06:42.421340 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:42.408752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49799 (local address 127.0.97.62:40535)
0620 03:06:42.408922 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:42.408927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:42.409309 (+ 382us) server_negotiation.cc:407] Connection header received
0620 03:06:42.409436 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:42.409439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:42.409506 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:42.409589 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:42.410301 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:42.411080 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:42.412320 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:42.412757 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:42.412861 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:42.421112 (+ 8251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49799: BlockingRecv error: recv got EOF from 127.0.97.3:49799 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 03:06:42 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:42 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:42.517346 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:42.509075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:34337)
0620 03:06:42.509664 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:42.509679 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:42.509847 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:42.510201 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:42.510206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:42.510621 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:42.510812 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:42.510821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:42.511886 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:42.511893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:42.512796 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:42.512807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:42.513507 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:42.513512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:42.513681 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:42.514294 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:42.514311 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:42.516980 (+ 2669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":255,"threads_started":1}
W20260620 03:06:42.517395 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:42.509260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34337 (local address 127.0.97.62:40535)
0620 03:06:42.509494 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:42.509500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:42.509799 (+ 299us) server_negotiation.cc:407] Connection header received
0620 03:06:42.509987 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:42.509992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:42.510062 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:42.510159 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:42.510952 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:42.511748 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:42.512935 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:42.513363 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:42.513493 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:42.517179 (+ 3686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34337: BlockingRecv error: recv got EOF from 127.0.97.1:34337 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 03:06:42 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:42 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:42.958384 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:42.950161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:46561)
0620 03:06:42.950708 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:42.950725 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:42.950845 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:42.951161 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:42.951164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:42.951618 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:42.951784 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:42.951792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:42.952824 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:42.952828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:42.953772 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:42.953785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:42.954456 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:42.954460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:42.954579 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:42.955139 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:42.955156 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:42.958079 (+ 2923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":240,"threads_started":1}
W20260620 03:06:42.958570 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:42.950201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46561 (local address 127.0.97.62:40535)
0620 03:06:42.950463 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:42.950472 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:42.950828 (+ 356us) server_negotiation.cc:407] Connection header received
0620 03:06:42.950945 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:42.950949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:42.951025 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:42.951108 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:42.951929 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:42.952676 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:42.953908 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:42.954331 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:42.954424 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:42.958276 (+ 3852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46561: BlockingRecv error: recv got EOF from 127.0.97.2:46561 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Jun 20 03:06:43 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:43 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:43.430116 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:43.422058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.3:44043)
0620 03:06:43.422668 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:43.422684 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:43.422795 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:43.423124 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:43.423126 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:43.423546 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:43.423711 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:43.423720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:43.424703 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:43.424706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:43.425581 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:43.425591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:43.426309 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:43.426314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:43.426418 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:43.427022 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:43.427040 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:43.429741 (+ 2701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":275,"threads_started":1}
W20260620 03:06:43.430140 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:43.422123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44043 (local address 127.0.97.62:40535)
0620 03:06:43.422312 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:43.422317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:43.422769 (+ 452us) server_negotiation.cc:407] Connection header received
0620 03:06:43.422912 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:43.422917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:43.422993 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:43.423089 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:43.423839 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:43.424588 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:43.425722 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:43.426176 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:43.426275 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:43.429943 (+ 3668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44043: BlockingRecv error: recv got EOF from 127.0.97.3:44043 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 03:06:43 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:43 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:43.526541 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:43.518309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:59361)
0620 03:06:43.518820 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:43.518844 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:43.518956 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:43.519261 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:43.519265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:43.519694 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:43.519844 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:43.519853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:43.520846 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:43.520851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:43.522019 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:43.522031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:43.522926 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:43.522932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:43.523089 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:43.523655 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:43.523681 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:43.526316 (+ 2635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":296,"threads_started":1}
W20260620 03:06:43.526739 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:43.518473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59361 (local address 127.0.97.62:40535)
0620 03:06:43.518629 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:43.518633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:43.518913 (+ 280us) server_negotiation.cc:407] Connection header received
0620 03:06:43.519056 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:43.519059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:43.519116 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:43.519214 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:43.519969 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:43.520700 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:43.522180 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:43.522747 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:43.522875 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:43.526501 (+ 3626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59361: BlockingRecv error: recv got EOF from 127.0.97.1:59361 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 03:06:43 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:43 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:43.968361 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:43.959428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:50635)
0620 03:06:43.959947 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:43.959963 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:43.960080 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:43.960366 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:43.960369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:43.960727 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:43.960867 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:43.960874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:43.962501 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:43.962505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:43.963436 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:43.963448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:43.964158 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:43.964162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:43.964269 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:43.964895 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:43.964912 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:43.968022 (+ 3110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":309,"threads_started":1}
W20260620 03:06:43.968417 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:43.959543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50635 (local address 127.0.97.62:40535)
0620 03:06:43.959703 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:43.959707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:43.960064 (+ 357us) server_negotiation.cc:407] Connection header received
0620 03:06:43.960183 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:43.960186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:43.960249 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:43.960324 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:43.961021 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:43.962350 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:43.963577 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:43.963981 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:43.964082 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:43.968204 (+ 4122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50635: BlockingRecv error: recv got EOF from 127.0.97.2:50635 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 03:06:44 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:44 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:44.440012 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:44.431179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.3:59301)
0620 03:06:44.431713 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:44.431728 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:44.431881 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:44.432236 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:44.432241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:44.432618 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:44.432756 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:44.432764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:44.433749 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:44.433755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:44.435201 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:44.435214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:44.436182 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:44.436187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:44.436298 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:44.436905 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:44.436921 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:44.439670 (+ 2749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":314,"threads_started":1}
W20260620 03:06:44.440116 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:44.431236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59301 (local address 127.0.97.62:40535)
0620 03:06:44.431527 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:44.431536 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:44.431813 (+ 277us) server_negotiation.cc:407] Connection header received
0620 03:06:44.431998 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:44.432004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:44.432097 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:44.432197 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:44.432892 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:44.433609 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:44.435374 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:44.435799 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:44.435927 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:44.439881 (+ 3954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59301: BlockingRecv error: recv got EOF from 127.0.97.3:59301 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Jun 20 03:06:44 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:44 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:44.535733 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:44.527442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:56281)
0620 03:06:44.527961 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:44.527978 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:44.528093 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:44.528389 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:44.528392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:44.528754 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:44.528914 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:44.528921 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:44.529867 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:44.529871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:44.530963 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:44.530976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:44.531963 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:44.531973 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:44.532135 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:44.532742 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:44.532759 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:44.535494 (+ 2735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":278,"threads_started":1}
W20260620 03:06:44.535888 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:44.527513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56281 (local address 127.0.97.62:40535)
0620 03:06:44.527700 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:44.527704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:44.528091 (+ 387us) server_negotiation.cc:407] Connection header received
0620 03:06:44.528213 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:44.528217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:44.528274 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:44.528357 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:44.529045 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:44.529749 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:44.531097 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:44.531793 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:44.531919 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:44.535685 (+ 3766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56281: BlockingRecv error: recv got EOF from 127.0.97.1:56281 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 03:06:44 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:44 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:44.978618 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:44.969458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:37565)
0620 03:06:44.970014 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:44.970030 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:44.970156 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:44.970459 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:44.970462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:44.970934 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:44.971111 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:44.971120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:44.972377 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:44.972383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:44.973609 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:44.973621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:44.974278 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:44.974283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:44.974428 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:44.974985 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:44.975001 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:44.978353 (+ 3352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":311,"threads_started":1}
W20260620 03:06:44.978746 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:44.969569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37565 (local address 127.0.97.62:40535)
0620 03:06:44.969753 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:44.969758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:44.970132 (+ 374us) server_negotiation.cc:407] Connection header received
0620 03:06:44.970260 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:44.970265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:44.970333 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:44.970416 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:44.971267 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:44.972263 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:44.973731 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:44.974133 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:44.974240 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:44.978529 (+ 4289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37565: BlockingRecv error: recv got EOF from 127.0.97.2:37565 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 03:06:45 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:45 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:45.449510 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:45.440964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.3:42919)
0620 03:06:45.441485 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:45.441501 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:45.441613 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:45.441892 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:45.441894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:45.442269 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:45.442439 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:45.442447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:45.443471 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:45.443475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:45.444516 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:45.444528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:45.445302 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:45.445307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:45.445479 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:45.446036 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:45.446053 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:45.449122 (+ 3069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":275,"threads_started":1}
W20260620 03:06:45.449622 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:45.441032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42919 (local address 127.0.97.62:40535)
0620 03:06:45.441192 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:45.441196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:45.441585 (+ 389us) server_negotiation.cc:407] Connection header received
0620 03:06:45.441705 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:45.441709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:45.441770 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:45.441860 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:45.442565 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:45.443345 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:45.444642 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:45.445108 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:45.445240 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:45.449352 (+ 4112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42919: BlockingRecv error: recv got EOF from 127.0.97.3:42919 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 03:06:45 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:45 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:45.545074 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:45.536673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:45389)
0620 03:06:45.537185 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:45.537200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:45.537362 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:45.537682 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:45.537686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:45.538079 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:45.538214 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:45.538222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:45.539200 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:45.539208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:45.540141 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:45.540152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:45.540875 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:45.540880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:45.541042 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:45.541609 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:45.541625 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:45.544753 (+ 3128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":291,"threads_started":1}
W20260620 03:06:45.545182 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:45.536854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45389 (local address 127.0.97.62:40535)
0620 03:06:45.537076 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:45.537082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:45.537295 (+ 213us) server_negotiation.cc:407] Connection header received
0620 03:06:45.537482 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:45.537487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:45.537552 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:45.537649 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:45.538357 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:45.539073 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:45.540286 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:45.540737 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:45.540865 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:45.544942 (+ 4077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45389: BlockingRecv error: recv got EOF from 127.0.97.1:45389 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 03:06:45 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:45 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:45.987375 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:45.979708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:50051)
0620 03:06:45.980218 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:45.980234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:45.980337 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:45.980658 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:45.980662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:45.981028 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:45.981170 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:45.981178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:45.982151 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:45.982156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:45.983062 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:45.983072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:45.983773 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:45.983777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:45.983884 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:45.984441 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:45.984457 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:45.987108 (+ 2651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":305,"threads_started":1}
W20260620 03:06:45.987612 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:45.979732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50051 (local address 127.0.97.62:40535)
0620 03:06:45.979901 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:45.979905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:45.980307 (+ 402us) server_negotiation.cc:407] Connection header received
0620 03:06:45.980461 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:45.980466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:45.980541 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:45.980624 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:45.981283 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:45.982005 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:45.983184 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:45.983646 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:45.983736 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:45.987312 (+ 3576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50051: BlockingRecv error: recv got EOF from 127.0.97.2:50051 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 03:06:46 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:46 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:46.462577 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:46.453920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.3:53803)
0620 03:06:46.454866 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:46.454884 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:46.455021 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:46.455500 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:46.455505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:46.455875 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:46.456016 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:46.456024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:46.457006 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:46.457017 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:46.457965 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:46.457976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:46.458670 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:46.458676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:46.458820 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:46.459620 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:46.459637 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:46.462237 (+ 2600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":634,"threads_started":1}
W20260620 03:06:46.462659 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:46.454029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53803 (local address 127.0.97.62:40535)
0620 03:06:46.454186 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:46.454192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:46.454963 (+ 771us) server_negotiation.cc:407] Connection header received
0620 03:06:46.455244 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:46.455249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:46.455362 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:46.455467 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:46.456152 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:46.456871 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:46.458106 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:46.458510 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:46.458629 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:46.462433 (+ 3804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53803: BlockingRecv error: recv got EOF from 127.0.97.3:53803 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 03:06:46 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:46 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:46.554592 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:46.546041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:42025)
0620 03:06:46.546587 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:46.546604 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:46.546754 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:46.547167 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:46.547172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:46.547634 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:46.547810 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:46.547818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:46.548968 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:46.548996 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:46.549938 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:46.549949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:46.550676 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:46.550681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:46.550866 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:46.551558 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:46.551575 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:46.554248 (+ 2673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":295,"threads_started":1}
W20260620 03:06:46.554682 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:46.546211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42025 (local address 127.0.97.62:40535)
0620 03:06:46.546449 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:46.546455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:46.546687 (+ 232us) server_negotiation.cc:407] Connection header received
0620 03:06:46.546924 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:46.546929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:46.546990 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:46.547100 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:46.547951 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:46.548844 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:46.550072 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:46.550519 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:46.550661 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:46.554437 (+ 3776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42025: BlockingRecv error: recv got EOF from 127.0.97.1:42025 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 03:06:46 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:46 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:46.996285 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:46.988325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:46557)
0620 03:06:46.988835 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:46.988851 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:46.988967 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:46.989247 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:46.989250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:46.989634 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:46.989797 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:46.989805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:46.990789 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:46.990794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:46.991770 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:46.991782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:46.992527 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:46.992532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:46.992671 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:46.993288 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:46.993311 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:46.996053 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":292,"threads_started":1}
W20260620 03:06:46.996526 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:46.988356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46557 (local address 127.0.97.62:40535)
0620 03:06:46.988554 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:46.988564 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:46.988925 (+ 361us) server_negotiation.cc:407] Connection header received
0620 03:06:46.989058 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:46.989062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:46.989137 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:46.989211 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:46.989910 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:46.990651 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:46.991914 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:46.992360 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:46.992461 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:46.996283 (+ 3822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46557: BlockingRecv error: recv got EOF from 127.0.97.2:46557 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 03:06:47 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:47 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:47.476361 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:47.463572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.3:43113)
0620 03:06:47.464065 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:47.464081 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:47.464197 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:47.464518 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:47.464521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:47.464905 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:47.465072 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:47.465080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:47.466054 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:47.466058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:47.466920 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:47.466929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:47.467591 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:47.467595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:47.467735 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:47.473126 (+ 5391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:47.473144 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:47.476016 (+ 2872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":288,"threads_started":1}
W20260620 03:06:47.476485 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:47.463686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43113 (local address 127.0.97.62:40535)
0620 03:06:47.463921 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:47.463928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:47.464186 (+ 258us) server_negotiation.cc:407] Connection header received
0620 03:06:47.464323 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:47.464327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:47.464390 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:47.464473 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:47.465224 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:47.465942 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:47.467042 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:47.467451 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:47.467537 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:47.476248 (+ 8711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43113: BlockingRecv error: recv got EOF from 127.0.97.3:43113 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 03:06:47 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:47 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:47.564102 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:47.555574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:37033)
0620 03:06:47.556131 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:47.556148 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:47.556300 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:47.556625 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:47.556628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:47.556969 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:47.557122 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:47.557130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:47.558242 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:47.558247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:47.559161 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:47.559172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:47.559962 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:47.559967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:47.560154 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:47.560811 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:47.560836 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:47.563877 (+ 3041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":320,"threads_started":1}
W20260620 03:06:47.564358 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:47.555629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37033 (local address 127.0.97.62:40535)
0620 03:06:47.555897 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:47.555908 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:47.556231 (+ 323us) server_negotiation.cc:407] Connection header received
0620 03:06:47.556435 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:47.556439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:47.556502 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:47.556590 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:47.557262 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:47.558052 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:47.559337 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:47.559817 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:47.559913 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:47.564068 (+ 4155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37033: BlockingRecv error: recv got EOF from 127.0.97.1:37033 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 03:06:48 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:48 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:48.005381 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:47.997361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:38429)
0620 03:06:47.997887 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:47.997903 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:47.998053 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:47.998395 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:47.998399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:47.998843 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:47.999004 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:47.999013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:47.999983 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:47.999987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:48.000932 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:48.000943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:48.001716 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:48.001720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:48.001854 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:48.002446 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:48.002464 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:48.005150 (+ 2686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":260,"threads_started":1}
W20260620 03:06:48.005561 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:47.997436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38429 (local address 127.0.97.62:40535)
0620 03:06:47.997677 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:47.997683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:47.997986 (+ 303us) server_negotiation.cc:407] Connection header received
0620 03:06:47.998187 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:47.998191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:47.998250 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:47.998361 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:47.999132 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:47.999871 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:48.001075 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:48.001595 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:48.001686 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:48.005335 (+ 3649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38429: BlockingRecv error: recv got EOF from 127.0.97.2:38429 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 03:06:48 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:48 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:48.485877 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:48.477305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.3:40269)
0620 03:06:48.477848 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:48.477864 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:48.477972 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:48.478271 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:48.478275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:48.478741 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:48.478895 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:48.478902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:48.480074 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:48.480078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:48.481011 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:48.481022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:48.481691 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:48.481695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:48.481824 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:48.482395 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:48.482412 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:48.485454 (+ 3042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":305,"threads_started":1}
W20260620 03:06:48.485972 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:48.477385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40269 (local address 127.0.97.62:40535)
0620 03:06:48.477559 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:48.477564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:48.477929 (+ 365us) server_negotiation.cc:407] Connection header received
0620 03:06:48.478060 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:48.478063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:48.478122 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:48.478201 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:48.479029 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:48.479930 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:48.481137 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:48.481563 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:48.481654 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:48.485705 (+ 4051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40269: BlockingRecv error: recv got EOF from 127.0.97.3:40269 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 03:06:48 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:48 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:48.573303 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:48.565273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:45665)
0620 03:06:48.565768 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:48.565784 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:48.565900 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:48.566210 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:48.566213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:48.566707 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:48.566879 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:48.566887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:48.568102 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:48.568106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:48.569008 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:48.569019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:48.569731 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:48.569735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:48.569846 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:48.570425 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:48.570441 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:48.573061 (+ 2620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":253,"threads_started":1}
W20260620 03:06:48.573560 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:48.565314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45665 (local address 127.0.97.62:40535)
0620 03:06:48.565546 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:48.565553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:48.565864 (+ 311us) server_negotiation.cc:407] Connection header received
0620 03:06:48.565999 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:48.566003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:48.566076 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:48.566197 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:48.567015 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:48.567978 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:48.569152 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:48.569562 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:48.569666 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:48.573280 (+ 3614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45665: BlockingRecv error: recv got EOF from 127.0.97.1:45665 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 03:06:49 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:49 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:49.015241 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:49.006374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:55645)
0620 03:06:49.007050 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:49.007067 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:49.007177 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:49.007571 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:49.007578 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:49.008038 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:49.008197 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:49.008207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:49.009396 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:49.009401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:49.010340 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:49.010353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:49.011182 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:49.011189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:49.011399 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:49.012015 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:49.012033 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:49.014863 (+ 2830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":406,"threads_started":1}
W20260620 03:06:49.015324 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:49.006553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55645 (local address 127.0.97.62:40535)
0620 03:06:49.006787 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:49.006794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:49.007138 (+ 344us) server_negotiation.cc:407] Connection header received
0620 03:06:49.007283 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:49.007288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:49.007391 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:49.007511 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:49.008359 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:49.009252 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:49.010485 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:49.010990 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:49.011091 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:49.015062 (+ 3971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55645: BlockingRecv error: recv got EOF from 127.0.97.2:55645 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 03:06:49 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:49 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:49.495617 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:49.486869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.3:58793)
0620 03:06:49.487454 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:49.487470 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:49.487622 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:49.487952 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:49.487957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:49.488440 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:49.488592 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:49.488600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:49.489639 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:49.489646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:49.490572 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:49.490582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:49.491340 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:49.491350 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:49.491515 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:49.492098 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:49.492122 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:49.495398 (+ 3276us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":292,"threads_started":1}
W20260620 03:06:49.495776 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:49.486915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58793 (local address 127.0.97.62:40535)
0620 03:06:49.487104 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:49.487108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:49.487564 (+ 456us) server_negotiation.cc:407] Connection header received
0620 03:06:49.487738 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:49.487748 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:49.487823 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:49.487945 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:49.488723 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:49.489516 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:49.490710 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:49.491125 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:49.491255 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:49.495553 (+ 4298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58793: BlockingRecv error: recv got EOF from 127.0.97.3:58793 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 03:06:49 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:49 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:49.582626 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:49.574245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:38639)
0620 03:06:49.575013 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:49.575032 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:49.575183 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:49.575584 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:49.575590 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:49.575972 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:49.576145 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:49.576154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:49.577155 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:49.577161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:49.578082 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:49.578092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:49.578759 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:49.578765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:49.578937 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:49.579549 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:49.579565 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:49.582385 (+ 2820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":486,"threads_started":1}
W20260620 03:06:49.582813 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:49.574409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38639 (local address 127.0.97.62:40535)
0620 03:06:49.574647 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:49.574655 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:49.575116 (+ 461us) server_negotiation.cc:407] Connection header received
0620 03:06:49.575358 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:49.575362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:49.575450 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:49.575551 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:49.576283 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:49.577019 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:49.578217 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:49.578619 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:49.578747 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:49.582587 (+ 3840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38639: BlockingRecv error: recv got EOF from 127.0.97.1:38639 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Jun 20 03:06:50 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:50 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:50.024338 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:50.016087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:46391)
0620 03:06:50.016595 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:50.016611 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:50.016717 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:50.016992 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:50.016995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:50.017359 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:50.017530 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:50.017538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:50.018580 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:50.018584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:50.019513 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:50.019524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:50.020336 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:50.020340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:50.020457 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:50.021030 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:50.021047 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:50.023970 (+ 2923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":276,"threads_started":1}
W20260620 03:06:50.024413 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:50.016243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46391 (local address 127.0.97.62:40535)
0620 03:06:50.016412 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:50.016416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:50.016688 (+ 272us) server_negotiation.cc:407] Connection header received
0620 03:06:50.016810 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:50.016814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:50.016872 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:50.016955 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:50.017695 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:50.018448 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:50.019673 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:50.020157 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:50.020276 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:50.024169 (+ 3893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46391: BlockingRecv error: recv got EOF from 127.0.97.2:46391 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 03:06:50 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:50 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:50.504572 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:50.496568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.3:33981)
0620 03:06:50.497099 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:50.497116 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:50.497227 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:50.497510 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:50.497514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:50.497851 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:50.498016 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:50.498024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:50.498976 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:50.498980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:50.499999 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:50.500010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:50.500759 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:50.500763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:50.500928 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:50.501511 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:50.501527 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:50.504215 (+ 2688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":295,"threads_started":1}
W20260620 03:06:50.504738 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:50.496588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33981 (local address 127.0.97.62:40535)
0620 03:06:50.496746 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:50.496750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:50.497185 (+ 435us) server_negotiation.cc:407] Connection header received
0620 03:06:50.497334 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:50.497338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:50.497397 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:50.497475 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:50.498129 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:50.498859 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:50.500120 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:50.500639 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:50.500725 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:50.504479 (+ 3754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33981: BlockingRecv error: recv got EOF from 127.0.97.3:33981 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 03:06:50 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:50 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:50.592628 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:50.583876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.1:44097)
0620 03:06:50.584418 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:50.584434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:50.584541 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:50.584868 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:50.584871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:50.585337 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:50.585528 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:50.585536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:50.586799 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:50.586803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:50.587930 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:50.587943 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:50.588635 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:50.588641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:50.588807 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:50.589402 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:50.589419 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:50.592246 (+ 2827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":313,"threads_started":1}
W20260620 03:06:50.592661 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:50.584080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44097 (local address 127.0.97.62:40535)
0620 03:06:50.584235 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:50.584239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:50.584501 (+ 262us) server_negotiation.cc:407] Connection header received
0620 03:06:50.584653 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:50.584656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:50.584728 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:50.584817 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:50.585708 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:50.586619 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:50.588063 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:50.588484 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:50.588623 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:50.592435 (+ 3812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44097: BlockingRecv error: recv got EOF from 127.0.97.1:44097 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 03:06:51 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:06:51 dist-test-slave-xv68 krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:06:51.034348 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:51.025596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40535 (local address 127.0.97.2:51459)
0620 03:06:51.026129 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:51.026150 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:51.026324 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:51.026904 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:51.026911 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:51.027644 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:51.027857 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:51.027868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:51.028980 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:51.028984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:51.029899 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:51.029910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:51.030660 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:51.030665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:51.030788 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:51.031428 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:51.031446 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:51.034057 (+ 2611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40535: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":302,"threads_started":1}
W20260620 03:06:51.034474 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:51.025683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51459 (local address 127.0.97.62:40535)
0620 03:06:51.025923 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:51.025930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:51.026268 (+ 338us) server_negotiation.cc:407] Connection header received
0620 03:06:51.026450 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:51.026455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:51.026550 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:51.026664 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:51.027995 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:51.028838 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:51.030024 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:51.030524 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:51.030625 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:51.034242 (+ 3617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51459: BlockingRecv error: recv got EOF from 127.0.97.2:51459 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 03:06:51.335522 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 9230
I20260620 03:06:51.347255 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:51.336455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.0.97.1:35601)
0620 03:06:51.336949 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:51.336963 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:51.337008 (+ 45us) server_negotiation.cc:407] Connection header received
0620 03:06:51.337071 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:51.337077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:51.337292 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:51.337445 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:51.338820 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:51.339860 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:51.341000 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:51.341520 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:51.341612 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:51.344101 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:51.344109 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:51.344121 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:51.344148 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:51.346810 (+ 2662us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 03:06:51.347039 (+ 229us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40182: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":242,"threads_started":1}
W20260620 03:06:51.347399 9690 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40182: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 03:06:51.347442 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 9230: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:35601: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 03:06:51.347676 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 9230
I20260620 03:06:51.364715 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 9381
I20260620 03:06:51.376020 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:51.365597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51828 (local address 127.0.97.2:41789)
0620 03:06:51.366148 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:51.366155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:51.366185 (+ 30us) server_negotiation.cc:407] Connection header received
0620 03:06:51.366241 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:51.366247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:51.366470 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:51.366617 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:51.367882 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:51.368964 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:51.370094 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:51.370640 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:51.370735 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:51.373164 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:51.373172 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:51.373185 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:51.373218 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:51.375542 (+ 2324us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 03:06:51.375786 (+ 244us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51828: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":259,"threads_started":1}
W20260620 03:06:51.376140 9691 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51828: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 03:06:51.376154 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 9381: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:41789: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 03:06:51.376289 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 9381
I20260620 03:06:51.393956 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 9512
I20260620 03:06:51.408547 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:51.395039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33912 (local address 127.0.97.3:46279)
0620 03:06:51.395767 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:51.395775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:51.395801 (+ 26us) server_negotiation.cc:407] Connection header received
0620 03:06:51.395857 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:51.395863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:51.396075 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:51.396280 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:51.398152 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:51.399892 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:51.401325 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:51.402257 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:51.402392 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:51.405502 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:51.405522 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:51.405538 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:51.405583 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:51.407874 (+ 2291us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 03:06:51.408194 (+ 320us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33912: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":408,"threads_started":1}
W20260620 03:06:51.408550 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 9512: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:46279: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 03:06:51.408650 9692 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33912: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 03:06:51.408705 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 9512
I20260620 03:06:51.425565 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 9161
I20260620 03:06:51.427134 9223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:51.542753 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 9161
2026-06-20T03:06:51Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21012 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T03:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:06:51Z Disabled control of system clock
I20260620 03:06:51.615281 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36809
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 03:06:51.929754 9697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:51.930045 9697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:51.930123 9697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:51.937260 9697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:06:51.937371 9697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:51.937409 9697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:51.937460 9697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:06:51.937505 9697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:06:51.946501 9697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36809
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:51.947762 9697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:51.949414 9697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:51.961372 9702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:51.961315 9705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:51.961285 9703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:51.961944 9697 server_base.cc:1034] running on GCE node
I20260620 03:06:51.962724 9697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:51.964098 9697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:51.965308 9697 hybrid_clock.cc:648] HybridClock initialized: now 1781924811965253 us; error 56 us; skew 500 ppm
I20260620 03:06:51.965657 9697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:06:51.967912 9697 webserver.cc:460] Webserver started at http://127.0.97.62:40547/ using document root <none> and password file <none>
I20260620 03:06:51.968576 9697 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:51.968660 9697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:51.968964 9697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:51.971791 9697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "1228c4c307e346189b8c9871cbc5f362"
format_stamp: "Formatted at 2026-06-20 03:06:51 on dist-test-slave-xv68"
server_key: "743e78763d0357bda48c3d927240ff38"
server_key_iv: "064c3e8a67438e0c1b7571748c0b4099"
server_key_version: "encryptionkey@0"
I20260620 03:06:51.972553 9697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "1228c4c307e346189b8c9871cbc5f362"
format_stamp: "Formatted at 2026-06-20 03:06:51 on dist-test-slave-xv68"
server_key: "743e78763d0357bda48c3d927240ff38"
server_key_iv: "064c3e8a67438e0c1b7571748c0b4099"
server_key_version: "encryptionkey@0"
I20260620 03:06:51.977428 9697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 03:06:51.980744 9711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:51.982098 9697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 03:06:51.982302 9697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "1228c4c307e346189b8c9871cbc5f362"
format_stamp: "Formatted at 2026-06-20 03:06:51 on dist-test-slave-xv68"
server_key: "743e78763d0357bda48c3d927240ff38"
server_key_iv: "064c3e8a67438e0c1b7571748c0b4099"
server_key_version: "encryptionkey@0"
I20260620 03:06:51.982518 9697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:51.997519 9697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:51.998273 9697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:51.998551 9697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:52.016633 9697 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36809
I20260620 03:06:52.016644 9762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36809 every 8 connection(s)
I20260620 03:06:52.018020 9697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:06:52.022507 9763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:52.024024 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 9697
I20260620 03:06:52.024202 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:06:52.024916 388 external_mini_cluster.cc:1428] Setting key 5e14525c17297d978ea617b8586ad512
I20260620 03:06:52.033969 9763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362: Bootstrap starting.
I20260620 03:06:52.037422 9763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:52.038240 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:52.026846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.0.97.62:36809)
0620 03:06:52.027894 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:52.027917 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:52.027957 (+ 40us) server_negotiation.cc:407] Connection header received
0620 03:06:52.029534 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:52.029555 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:52.029983 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:52.030300 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:52.030862 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:52.032195 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:52.033999 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:52.034699 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:52.034821 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:52.036058 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:52.036065 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:52.036077 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:52.036587 (+ 510us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:52.036669 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:52.036689 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:52.036772 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:52.036989 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 03:06:52.037304 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":415,"threads_started":1}
I20260620 03:06:52.038486 9763 log.cc:824] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:52.041324 9763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362: No bootstrap required, opened a new log
I20260620 03:06:52.049326 9763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1228c4c307e346189b8c9871cbc5f362" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36809 } }
I20260620 03:06:52.049757 9763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:52.049868 9763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1228c4c307e346189b8c9871cbc5f362, State: Initialized, Role: FOLLOWER
I20260620 03:06:52.050433 9763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1228c4c307e346189b8c9871cbc5f362" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36809 } }
I20260620 03:06:52.050618 9763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:06:52.050714 9763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:06:52.050829 9763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:52.052867 9763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1228c4c307e346189b8c9871cbc5f362" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36809 } }
I20260620 03:06:52.053297 9763 leader_election.cc:302] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1228c4c307e346189b8c9871cbc5f362; no voters:
I20260620 03:06:52.053934 9763 leader_election.cc:288] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:06:52.054062 9768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:52.054829 9768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362 [term 1 LEADER]: Becoming Leader. State: Replica: 1228c4c307e346189b8c9871cbc5f362, State: Running, Role: LEADER
I20260620 03:06:52.055399 9768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1228c4c307e346189b8c9871cbc5f362" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36809 } }
I20260620 03:06:52.056002 9763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:06:52.057992 9769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1228c4c307e346189b8c9871cbc5f362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1228c4c307e346189b8c9871cbc5f362" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36809 } } }
I20260620 03:06:52.057986 9770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1228c4c307e346189b8c9871cbc5f362. Latest consensus state: current_term: 1 leader_uuid: "1228c4c307e346189b8c9871cbc5f362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1228c4c307e346189b8c9871cbc5f362" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36809 } } }
I20260620 03:06:52.058415 9769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:52.058413 9770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:52.068410 9773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:06:52.072566 9773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:06:52.079916 9773 catalog_manager.cc:1347] Generated new cluster ID: 78c69e2f5cb14f8697e49f61ed1f54ad
I20260620 03:06:52.080029 9773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:06:52.100844 9773 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:06:52.102646 9773 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:06:52.115536 9773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1228c4c307e346189b8c9871cbc5f362: Generated new TSK 0
I20260620 03:06:52.116519 9773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:06:52.134843 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36809
--builtin_ntp_servers=127.0.97.20:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 03:06:52.455997 9787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:52.456295 9787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:52.456380 9787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:52.463543 9787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:52.463639 9787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:52.463742 9787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:06:52.472651 9787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:52.473904 9787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:52.475579 9787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:52.487887 9792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:52.488039 9793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:52.488065 9795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:52.489095 9787 server_base.cc:1034] running on GCE node
I20260620 03:06:52.489719 9787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:52.490618 9787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:52.491839 9787 hybrid_clock.cc:648] HybridClock initialized: now 1781924812491781 us; error 38 us; skew 500 ppm
I20260620 03:06:52.492157 9787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:06:52.497534 9787 webserver.cc:460] Webserver started at http://127.0.97.1:39259/ using document root <none> and password file <none>
I20260620 03:06:52.498282 9787 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:52.498409 9787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:52.498765 9787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:52.501843 9787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "881e5e73386047b3b889ffcb279197fa"
format_stamp: "Formatted at 2026-06-20 03:06:52 on dist-test-slave-xv68"
server_key: "3fcb43fc1ac9e2c5a2f6445cda6628be"
server_key_iv: "00bc24cd33d96f7f99481fbd05ec3109"
server_key_version: "encryptionkey@0"
I20260620 03:06:52.502660 9787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "881e5e73386047b3b889ffcb279197fa"
format_stamp: "Formatted at 2026-06-20 03:06:52 on dist-test-slave-xv68"
server_key: "3fcb43fc1ac9e2c5a2f6445cda6628be"
server_key_iv: "00bc24cd33d96f7f99481fbd05ec3109"
server_key_version: "encryptionkey@0"
I20260620 03:06:52.508165 9787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 03:06:52.511633 9801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:52.513075 9787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:06:52.513315 9787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "881e5e73386047b3b889ffcb279197fa"
format_stamp: "Formatted at 2026-06-20 03:06:52 on dist-test-slave-xv68"
server_key: "3fcb43fc1ac9e2c5a2f6445cda6628be"
server_key_iv: "00bc24cd33d96f7f99481fbd05ec3109"
server_key_version: "encryptionkey@0"
I20260620 03:06:52.513561 9787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:52.537768 9787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:52.538731 9787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:52.539040 9787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:52.540102 9787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:52.541592 9787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:52.541692 9787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:52.541782 9787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:52.541843 9787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:52.577339 9787 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46417
I20260620 03:06:52.577431 9913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46417 every 8 connection(s)
I20260620 03:06:52.578656 9787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:06:52.585529 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 9787
I20260620 03:06:52.585728 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:06:52.586603 388 external_mini_cluster.cc:1428] Setting key 15e169d630e3c8ef88dc6e76f04c0294
I20260620 03:06:52.589362 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36809
--builtin_ntp_servers=127.0.97.20:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 03:06:52.591555 9916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:52.581197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44961 (local address 127.0.97.62:36809)
0620 03:06:52.581651 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:52.581656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:52.582867 (+ 1211us) server_negotiation.cc:407] Connection header received
0620 03:06:52.584741 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:52.584746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:52.584811 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:52.584907 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:52.586128 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:52.587363 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:52.588873 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:52.589272 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:52.589397 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:52.590692 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:52.590699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:52.590703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:52.590729 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:52.590766 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:52.590778 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:52.590874 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:52.591086 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 03:06:52.591174 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":191,"threads_started":1}
I20260620 03:06:52.592437 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:52.581778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36809 (local address 127.0.97.1:44961)
0620 03:06:52.582715 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:52.582755 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:52.584479 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:52.585135 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:52.585143 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:52.585273 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:52.585966 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:52.585986 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:52.587524 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:52.587529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:52.588730 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:52.588741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:52.589492 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:52.589500 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:52.589718 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:52.590401 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:06:52.590404 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:52.590441 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:06:52.590514 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:52.590886 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:52.590898 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:52.590906 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 03:06:52.591079 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 03:06:52.591478 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":281,"threads_started":1}
I20260620 03:06:52.593966 9914 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36809
I20260620 03:06:52.594259 9914 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:52.594935 9914 heartbeater.cc:511] Master 127.0.97.62:36809 requested a full tablet report, sending...
I20260620 03:06:52.596895 9728 ts_manager.cc:194] Registered new tserver with Master: 881e5e73386047b3b889ffcb279197fa (127.0.97.1:46417)
I20260620 03:06:52.598304 9728 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:44961
W20260620 03:06:52.905767 9918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:52.906075 9918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:52.906226 9918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:52.913321 9918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:52.913447 9918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:52.913611 9918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:06:52.922685 9918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:52.923972 9918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:52.925689 9918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:52.940250 9923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:52.940418 9924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:52.941064 9926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:52.941859 9918 server_base.cc:1034] running on GCE node
I20260620 03:06:52.942416 9918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:52.943280 9918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:52.944542 9918 hybrid_clock.cc:648] HybridClock initialized: now 1781924812944495 us; error 41 us; skew 500 ppm
I20260620 03:06:52.944901 9918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:06:52.947014 9918 webserver.cc:460] Webserver started at http://127.0.97.2:38527/ using document root <none> and password file <none>
I20260620 03:06:52.947772 9918 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:52.947899 9918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:52.948221 9918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:52.950979 9918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "19e048fbf4dd416baa4d3f7feeeb81b6"
format_stamp: "Formatted at 2026-06-20 03:06:52 on dist-test-slave-xv68"
server_key: "b24d7031caf47acd6476dadfd4a42898"
server_key_iv: "21b323e264e31a34fe7fd6730484e2fa"
server_key_version: "encryptionkey@0"
I20260620 03:06:52.951789 9918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "19e048fbf4dd416baa4d3f7feeeb81b6"
format_stamp: "Formatted at 2026-06-20 03:06:52 on dist-test-slave-xv68"
server_key: "b24d7031caf47acd6476dadfd4a42898"
server_key_iv: "21b323e264e31a34fe7fd6730484e2fa"
server_key_version: "encryptionkey@0"
I20260620 03:06:52.956861 9918 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:06:52.960253 9932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:52.961683 9918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:06:52.961884 9918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "19e048fbf4dd416baa4d3f7feeeb81b6"
format_stamp: "Formatted at 2026-06-20 03:06:52 on dist-test-slave-xv68"
server_key: "b24d7031caf47acd6476dadfd4a42898"
server_key_iv: "21b323e264e31a34fe7fd6730484e2fa"
server_key_version: "encryptionkey@0"
I20260620 03:06:52.962103 9918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:52.989804 9918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:52.990674 9918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:52.990943 9918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:52.991906 9918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:52.993342 9918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:52.993482 9918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:52.993613 9918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:52.993702 9918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:53.026430 9918 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33091
I20260620 03:06:53.026474 10044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33091 every 8 connection(s)
I20260620 03:06:53.027719 9918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:06:53.031471 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 9918
I20260620 03:06:53.031620 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:06:53.032104 388 external_mini_cluster.cc:1428] Setting key 98675a1be0de50e74e5cf0f5fe8e02b2
I20260620 03:06:53.034960 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36809
--builtin_ntp_servers=127.0.97.20:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 03:06:53.041019 9916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:53.030221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36525 (local address 127.0.97.62:36809)
0620 03:06:53.030427 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:53.030438 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:53.031925 (+ 1487us) server_negotiation.cc:407] Connection header received
0620 03:06:53.034069 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:53.034074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:53.034153 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:53.034258 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:53.035321 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:53.036518 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:53.037976 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:53.038651 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:53.038784 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:53.040055 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:53.040064 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:53.040071 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:53.040095 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:53.040136 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:53.040153 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:53.040267 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:53.040538 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 03:06:53.040638 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 03:06:53.041916 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:53.030717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36809 (local address 127.0.97.2:36525)
0620 03:06:53.031773 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:53.031815 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:53.033746 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:53.034435 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:53.034442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:53.034528 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:53.035059 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:53.035101 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:53.036658 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:53.036662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:53.037832 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:53.037843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:53.038847 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:53.038852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:53.039055 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:53.039767 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:06:53.039770 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:53.039810 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:06:53.039885 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:53.040319 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:53.040328 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:53.040336 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 03:06:53.040508 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 03:06:53.040825 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":335,"threads_started":1}
I20260620 03:06:53.043368 10045 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36809
I20260620 03:06:53.043664 10045 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:53.044353 10045 heartbeater.cc:511] Master 127.0.97.62:36809 requested a full tablet report, sending...
I20260620 03:06:53.045673 9728 ts_manager.cc:194] Registered new tserver with Master: 19e048fbf4dd416baa4d3f7feeeb81b6 (127.0.97.2:33091)
I20260620 03:06:53.046437 9728 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:36525
W20260620 03:06:53.357059 10048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:53.357386 10048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:53.357549 10048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:53.365127 10048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:53.365291 10048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:53.365468 10048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:06:53.374500 10048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:53.375829 10048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:53.377553 10048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:53.390132 10053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:53.390157 10056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:53.390275 10054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:53.400023 10048 server_base.cc:1034] running on GCE node
I20260620 03:06:53.400960 10048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:53.402319 10048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:53.403548 10048 hybrid_clock.cc:648] HybridClock initialized: now 1781924813403502 us; error 37 us; skew 500 ppm
I20260620 03:06:53.403993 10048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:06:53.406752 10048 webserver.cc:460] Webserver started at http://127.0.97.3:40251/ using document root <none> and password file <none>
I20260620 03:06:53.407480 10048 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:53.407568 10048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:53.407824 10048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:53.410549 10048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "490dbce1005b49148162d5c65a982e4e"
format_stamp: "Formatted at 2026-06-20 03:06:53 on dist-test-slave-xv68"
server_key: "6673a3ef2f100a9f23afe1ecb2fbaf59"
server_key_iv: "9a87295d32f9f267158a6fcc0accfd8d"
server_key_version: "encryptionkey@0"
I20260620 03:06:53.411263 10048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "490dbce1005b49148162d5c65a982e4e"
format_stamp: "Formatted at 2026-06-20 03:06:53 on dist-test-slave-xv68"
server_key: "6673a3ef2f100a9f23afe1ecb2fbaf59"
server_key_iv: "9a87295d32f9f267158a6fcc0accfd8d"
server_key_version: "encryptionkey@0"
I20260620 03:06:53.416306 10048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 03:06:53.419723 10062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:53.421005 10048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 03:06:53.421135 10048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "490dbce1005b49148162d5c65a982e4e"
format_stamp: "Formatted at 2026-06-20 03:06:53 on dist-test-slave-xv68"
server_key: "6673a3ef2f100a9f23afe1ecb2fbaf59"
server_key_iv: "9a87295d32f9f267158a6fcc0accfd8d"
server_key_version: "encryptionkey@0"
I20260620 03:06:53.421330 10048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:53.455137 10048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:53.456005 10048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:53.456291 10048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:53.457321 10048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:53.458765 10048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:53.458854 10048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:53.458977 10048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:53.459054 10048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:53.491338 10048 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44163
I20260620 03:06:53.491441 10174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44163 every 8 connection(s)
I20260620 03:06:53.492624 10048 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:06:53.497541 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 10048
I20260620 03:06:53.497702 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:06:53.498230 388 external_mini_cluster.cc:1428] Setting key 4c5989c5053a20b50985cbc698d18573
I20260620 03:06:53.505854 9916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:53.495387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41035 (local address 127.0.97.62:36809)
0620 03:06:53.495602 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:53.495609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:53.497165 (+ 1556us) server_negotiation.cc:407] Connection header received
0620 03:06:53.499604 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:53.499607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:53.499670 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:53.499758 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:53.500956 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:53.501672 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:53.503217 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:53.503714 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:53.503846 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:53.505063 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:53.505069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:53.505074 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:53.505100 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:53.505126 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:53.505137 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:53.505230 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:53.505461 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 03:06:53.505545 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 03:06:53.506821 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:53.496018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36809 (local address 127.0.97.3:41035)
0620 03:06:53.496983 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:53.497019 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:53.499185 (+ 2166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:53.500052 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:53.500064 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:53.500178 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:53.500811 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:53.500840 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:53.501820 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:53.501825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:53.503065 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:53.503081 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:53.503926 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:53.503934 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:53.504164 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:53.504773 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:06:53.504775 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:53.504811 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:06:53.504883 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:53.505228 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:53.505240 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:53.505249 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 03:06:53.505455 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 03:06:53.505818 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":330,"threads_started":1}
I20260620 03:06:53.508000 10175 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36809
I20260620 03:06:53.508288 10175 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:53.509011 10175 heartbeater.cc:511] Master 127.0.97.62:36809 requested a full tablet report, sending...
I20260620 03:06:53.510316 9728 ts_manager.cc:194] Registered new tserver with Master: 490dbce1005b49148162d5c65a982e4e (127.0.97.3:44163)
I20260620 03:06:53.511080 9728 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:41035
I20260620 03:06:53.513262 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 03:06:53.519385 9916 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:53.517824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33426 (local address 127.0.97.62:36809)
0620 03:06:53.517991 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:53.517995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:53.518470 (+ 475us) server_negotiation.cc:407] Connection header received
0620 03:06:53.518612 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:53.518616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:53.518679 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:53.518757 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:53.519099 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33426: BlockingRecv error: recv got EOF from 127.0.0.1:33426 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 03:06:53.519853 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 9787
I20260620 03:06:53.527571 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:53.520761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51686 (local address 127.0.97.1:46417)
0620 03:06:53.521276 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:53.521281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:53.521306 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:06:53.521354 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:53.521359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:53.521521 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:53.521634 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:53.521976 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:53.522940 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:53.524394 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:53.524921 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:53.525034 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:53.526162 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:53.526177 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:53.526188 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:53.526722 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:53.526775 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:53.526793 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:53.526915 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:53.527078 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 03:06:53.527195 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":265,"threads_started":1}
I20260620 03:06:53.528754 9909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:53.609099 9914 heartbeater.cc:503] Master 127.0.97.62:36809 was elected leader, sending a full tablet report...
I20260620 03:06:53.611481 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 9787
I20260620 03:06:53.627640 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 9918
I20260620 03:06:53.635195 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:53.628552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.0.97.2:33091)
0620 03:06:53.629022 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:53.629029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:53.629054 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:06:53.629117 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:53.629123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:53.629363 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:53.629487 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:53.629907 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:53.630891 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:53.632173 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:53.632730 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:53.632827 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:53.633837 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:53.633844 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:53.633857 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:53.634422 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:53.634464 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:53.634483 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:53.634580 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:53.634695 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 03:06:53.634832 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":226,"threads_started":1}
I20260620 03:06:53.636415 10040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:53.717010 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 9918
I20260620 03:06:53.734047 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 10048
I20260620 03:06:53.741971 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:53.735006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.0.97.3:44163)
0620 03:06:53.735637 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:53.735644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:53.735672 (+ 28us) server_negotiation.cc:407] Connection header received
0620 03:06:53.735726 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:53.735732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:53.735946 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:53.736117 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:53.736538 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:53.737543 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:53.738782 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:53.739397 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:53.739524 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:53.740566 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:53.740575 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:53.740587 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:53.741156 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:53.741199 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:53.741218 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:53.741341 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:53.741492 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 03:06:53.741623 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":360,"threads_started":1}
I20260620 03:06:53.743160 10170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:53.823401 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 10048
I20260620 03:06:53.838879 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 9697
I20260620 03:06:53.840071 9758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:53.940451 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 9697
2026-06-20T03:06:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2393 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T03:06:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:06:53Z Disabled control of system clock
I20260620 03:06:54.010826 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38259
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 03:06:54.327776 10194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:54.328051 10194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:54.328133 10194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:54.335094 10194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:06:54.335196 10194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:54.335232 10194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:54.335279 10194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:06:54.335345 10194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:06:54.344085 10194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38259
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:54.345239 10194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:54.346884 10194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:54.358088 10199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:54.358167 10200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:54.358163 10202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:54.359172 10194 server_base.cc:1034] running on GCE node
I20260620 03:06:54.359935 10194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:54.361194 10194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:54.362427 10194 hybrid_clock.cc:648] HybridClock initialized: now 1781924814362390 us; error 52 us; skew 500 ppm
I20260620 03:06:54.362758 10194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:06:54.364912 10194 webserver.cc:460] Webserver started at http://127.0.97.62:37011/ using document root <none> and password file <none>
I20260620 03:06:54.365578 10194 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:54.365706 10194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:54.366015 10194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:54.368759 10194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "9dd3743d1a424c80940ad8d8364f4139"
format_stamp: "Formatted at 2026-06-20 03:06:54 on dist-test-slave-xv68"
server_key: "8ba88b8ac5b091329248c66e6b36c41b"
server_key_iv: "f0b9b073f1e34d7dd3261cb78d3be704"
server_key_version: "encryptionkey@0"
I20260620 03:06:54.369524 10194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "9dd3743d1a424c80940ad8d8364f4139"
format_stamp: "Formatted at 2026-06-20 03:06:54 on dist-test-slave-xv68"
server_key: "8ba88b8ac5b091329248c66e6b36c41b"
server_key_iv: "f0b9b073f1e34d7dd3261cb78d3be704"
server_key_version: "encryptionkey@0"
I20260620 03:06:54.374186 10194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 03:06:54.377369 10208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:54.378623 10194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:06:54.378791 10194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "9dd3743d1a424c80940ad8d8364f4139"
format_stamp: "Formatted at 2026-06-20 03:06:54 on dist-test-slave-xv68"
server_key: "8ba88b8ac5b091329248c66e6b36c41b"
server_key_iv: "f0b9b073f1e34d7dd3261cb78d3be704"
server_key_version: "encryptionkey@0"
I20260620 03:06:54.378997 10194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:54.384516 10194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:54.385273 10194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:54.385572 10194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:54.402665 10194 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38259
I20260620 03:06:54.402697 10259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38259 every 8 connection(s)
I20260620 03:06:54.404006 10194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:06:54.407625 10260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:54.409973 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 10194
I20260620 03:06:54.410180 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:06:54.410959 388 external_mini_cluster.cc:1428] Setting key a182a1a0ef9abb18b862ec44411cee31
I20260620 03:06:54.419072 10263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:54.413077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.0.97.62:38259)
0620 03:06:54.414082 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:54.414092 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:54.414131 (+ 39us) server_negotiation.cc:407] Connection header received
0620 03:06:54.415730 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:54.415751 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:54.416142 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:54.416544 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:54.417583 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:54.417589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:54.417604 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:54.417620 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:54.417674 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:54.417687 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:54.417920 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:54.418042 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 03:06:54.418252 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":355,"threads_started":1}
I20260620 03:06:54.419590 10260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139: Bootstrap starting.
I20260620 03:06:54.422904 10260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:54.423982 10260 log.cc:824] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:54.426733 10260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139: No bootstrap required, opened a new log
I20260620 03:06:54.434043 10260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dd3743d1a424c80940ad8d8364f4139" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38259 } }
I20260620 03:06:54.434396 10260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:54.434499 10260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd3743d1a424c80940ad8d8364f4139, State: Initialized, Role: FOLLOWER
I20260620 03:06:54.435038 10260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dd3743d1a424c80940ad8d8364f4139" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38259 } }
I20260620 03:06:54.435331 10260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:06:54.435484 10260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:06:54.435663 10260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:54.437677 10260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dd3743d1a424c80940ad8d8364f4139" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38259 } }
I20260620 03:06:54.438146 10260 leader_election.cc:302] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dd3743d1a424c80940ad8d8364f4139; no voters:
I20260620 03:06:54.438773 10260 leader_election.cc:288] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:06:54.439015 10265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:54.439801 10265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139 [term 1 LEADER]: Becoming Leader. State: Replica: 9dd3743d1a424c80940ad8d8364f4139, State: Running, Role: LEADER
I20260620 03:06:54.440295 10265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dd3743d1a424c80940ad8d8364f4139" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38259 } }
I20260620 03:06:54.440887 10260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:06:54.442947 10267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dd3743d1a424c80940ad8d8364f4139. Latest consensus state: current_term: 1 leader_uuid: "9dd3743d1a424c80940ad8d8364f4139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dd3743d1a424c80940ad8d8364f4139" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38259 } } }
I20260620 03:06:54.443264 10267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:54.443486 10266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dd3743d1a424c80940ad8d8364f4139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dd3743d1a424c80940ad8d8364f4139" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38259 } } }
I20260620 03:06:54.443989 10266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139 [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:54.445302 10270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:06:54.450383 10270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:06:54.458037 10270 catalog_manager.cc:1347] Generated new cluster ID: 282b01c4bccb446d8326d6723f24cdc0
I20260620 03:06:54.458158 10270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:06:54.483741 10270 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:06:54.484236 10270 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:06:54.501668 10270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dd3743d1a424c80940ad8d8364f4139: Generated new TSK 0
I20260620 03:06:54.502738 10270 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:06:54.518018 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38259
--builtin_ntp_servers=127.0.97.20:39517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 03:06:54.845743 10284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:54.846084 10284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:54.846261 10284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:54.853440 10284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:54.853535 10284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:54.853710 10284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:06:54.862646 10284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:54.863919 10284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:54.865645 10284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:54.878556 10290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:54.878623 10289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:54.879331 10292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:54.879457 10284 server_base.cc:1034] running on GCE node
I20260620 03:06:54.880089 10284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:54.881006 10284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:54.882222 10284 hybrid_clock.cc:648] HybridClock initialized: now 1781924814882147 us; error 45 us; skew 500 ppm
I20260620 03:06:54.882609 10284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:06:54.884892 10284 webserver.cc:460] Webserver started at http://127.0.97.1:35709/ using document root <none> and password file <none>
I20260620 03:06:54.885577 10284 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:54.885686 10284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:54.885998 10284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:54.890218 10284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "d320853770a74132a83246b18945d70e"
format_stamp: "Formatted at 2026-06-20 03:06:54 on dist-test-slave-xv68"
server_key: "8f0fb3deec0dfc1efdde85c88174ed50"
server_key_iv: "d7e15fd2bcee82823be7da4ac29d9e13"
server_key_version: "encryptionkey@0"
I20260620 03:06:54.890997 10284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d320853770a74132a83246b18945d70e"
format_stamp: "Formatted at 2026-06-20 03:06:54 on dist-test-slave-xv68"
server_key: "8f0fb3deec0dfc1efdde85c88174ed50"
server_key_iv: "d7e15fd2bcee82823be7da4ac29d9e13"
server_key_version: "encryptionkey@0"
I20260620 03:06:54.896059 10284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 03:06:54.899504 10298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:54.900887 10284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:06:54.901043 10284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "d320853770a74132a83246b18945d70e"
format_stamp: "Formatted at 2026-06-20 03:06:54 on dist-test-slave-xv68"
server_key: "8f0fb3deec0dfc1efdde85c88174ed50"
server_key_iv: "d7e15fd2bcee82823be7da4ac29d9e13"
server_key_version: "encryptionkey@0"
I20260620 03:06:54.901271 10284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:54.906546 10284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:54.907346 10284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:54.907608 10284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:54.908474 10284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:54.909765 10284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:54.909883 10284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:54.910004 10284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:54.910085 10284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:54.940965 10284 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42589
I20260620 03:06:54.940990 10410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42589 every 8 connection(s)
I20260620 03:06:54.942437 10284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:06:54.950699 10413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:54.944945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55217 (local address 127.0.97.62:38259)
0620 03:06:54.945394 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:54.945401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:54.946739 (+ 1338us) server_negotiation.cc:407] Connection header received
0620 03:06:54.948464 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:54.948469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:54.948534 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:54.948655 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:54.949959 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:54.949965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:54.949969 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:54.949988 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:54.950034 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:54.950046 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:54.950167 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:54.950360 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 03:06:54.950426 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":221,"threads_started":1}
I20260620 03:06:54.951673 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:54.945616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38259 (local address 127.0.97.1:55217)
0620 03:06:54.946571 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:54.946606 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:54.948203 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:54.948887 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:54.948896 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:54.948986 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:54.949669 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:06:54.949671 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:54.949707 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:06:54.949787 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:54.950138 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:54.950148 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:54.950155 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 03:06:54.950436 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 03:06:54.950671 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":302,"threads_started":1}
I20260620 03:06:54.952056 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 10284
I20260620 03:06:54.952273 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:06:54.953109 388 external_mini_cluster.cc:1428] Setting key a52599f4c627d634d7f4afe2ab5ec77a
I20260620 03:06:54.953347 10411 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38259
I20260620 03:06:54.953641 10411 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:54.954278 10411 heartbeater.cc:511] Master 127.0.97.62:38259 requested a full tablet report, sending...
I20260620 03:06:54.955974 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38259
--builtin_ntp_servers=127.0.97.20:39517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 03:06:54.956457 10225 ts_manager.cc:194] Registered new tserver with Master: d320853770a74132a83246b18945d70e (127.0.97.1:42589)
W20260620 03:06:55.277757 10415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:55.278048 10415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:55.278139 10415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:55.285398 10415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:55.285494 10415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:55.285600 10415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:06:55.294360 10415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:55.295574 10415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:55.297199 10415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:55.312839 10420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:55.313014 10421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:55.313848 10423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:55.314605 10415 server_base.cc:1034] running on GCE node
I20260620 03:06:55.315237 10415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:55.316586 10415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:55.317796 10415 hybrid_clock.cc:648] HybridClock initialized: now 1781924815317719 us; error 52 us; skew 500 ppm
I20260620 03:06:55.318078 10415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:06:55.321038 10415 webserver.cc:460] Webserver started at http://127.0.97.2:33781/ using document root <none> and password file <none>
I20260620 03:06:55.322034 10415 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:55.322161 10415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:55.322507 10415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:55.326503 10415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "90c13d59d224482da9332b7abd603c0d"
format_stamp: "Formatted at 2026-06-20 03:06:55 on dist-test-slave-xv68"
server_key: "02fbbf16a83ed1838b07f8a351851594"
server_key_iv: "d9f4fc5bd8e6a2df87bfff3a16739b58"
server_key_version: "encryptionkey@0"
I20260620 03:06:55.327633 10415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "90c13d59d224482da9332b7abd603c0d"
format_stamp: "Formatted at 2026-06-20 03:06:55 on dist-test-slave-xv68"
server_key: "02fbbf16a83ed1838b07f8a351851594"
server_key_iv: "d9f4fc5bd8e6a2df87bfff3a16739b58"
server_key_version: "encryptionkey@0"
I20260620 03:06:55.333477 10415 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:06:55.337888 10429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:55.339493 10415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:06:55.339655 10415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "90c13d59d224482da9332b7abd603c0d"
format_stamp: "Formatted at 2026-06-20 03:06:55 on dist-test-slave-xv68"
server_key: "02fbbf16a83ed1838b07f8a351851594"
server_key_iv: "d9f4fc5bd8e6a2df87bfff3a16739b58"
server_key_version: "encryptionkey@0"
I20260620 03:06:55.339874 10415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:55.345598 10415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:55.346619 10415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:55.346868 10415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:55.347893 10415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:55.349751 10415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:55.349895 10415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:55.349998 10415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:55.350073 10415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:55.381922 10415 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45707
I20260620 03:06:55.382027 10541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45707 every 8 connection(s)
I20260620 03:06:55.383410 10415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:06:55.386886 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 10415
I20260620 03:06:55.387079 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:06:55.387632 388 external_mini_cluster.cc:1428] Setting key 28d1953c8214fba9a12dd2897baf3fbe
I20260620 03:06:55.390580 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38259
--builtin_ntp_servers=127.0.97.20:39517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 03:06:55.391713 10413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:55.385894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55129 (local address 127.0.97.62:38259)
0620 03:06:55.386076 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:55.386080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:55.387600 (+ 1520us) server_negotiation.cc:407] Connection header received
0620 03:06:55.389409 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:55.389414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:55.389474 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:55.389553 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:55.390875 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:55.390879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:55.390882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:55.390897 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:55.390922 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:55.390933 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:55.390998 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:55.391268 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 03:06:55.391357 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 03:06:55.392452 10544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:55.386337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38259 (local address 127.0.97.2:55129)
0620 03:06:55.387433 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:55.387476 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:55.389099 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:55.389748 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:55.389757 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:55.389843 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:55.390537 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:06:55.390539 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:55.390575 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:06:55.390654 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:55.391067 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:55.391074 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:55.391079 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 03:06:55.391275 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 03:06:55.391540 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":364,"threads_started":1}
I20260620 03:06:55.393734 10542 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38259
I20260620 03:06:55.394021 10542 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:55.394690 10542 heartbeater.cc:511] Master 127.0.97.62:38259 requested a full tablet report, sending...
I20260620 03:06:55.396531 10225 ts_manager.cc:194] Registered new tserver with Master: 90c13d59d224482da9332b7abd603c0d (127.0.97.2:45707)
W20260620 03:06:55.710556 10545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:55.710850 10545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:55.711024 10545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:55.718178 10545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:55.718314 10545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:55.718502 10545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:06:55.727451 10545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:55.728703 10545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:55.730444 10545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:55.743424 10550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:55.743469 10551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:55.743494 10553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:55.744455 10545 server_base.cc:1034] running on GCE node
I20260620 03:06:55.744972 10545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:55.745895 10545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:55.747079 10545 hybrid_clock.cc:648] HybridClock initialized: now 1781924815747026 us; error 55 us; skew 500 ppm
I20260620 03:06:55.747392 10545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:06:55.749413 10545 webserver.cc:460] Webserver started at http://127.0.97.3:33455/ using document root <none> and password file <none>
I20260620 03:06:55.750000 10545 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:55.750080 10545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:55.750321 10545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:55.753010 10545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "40154f9f64cd40228d85a33bd98028cb"
format_stamp: "Formatted at 2026-06-20 03:06:55 on dist-test-slave-xv68"
server_key: "54ab937695ef1d18f8c805e04e654d39"
server_key_iv: "9922fbe6fa676f73288e7697742bcfc2"
server_key_version: "encryptionkey@0"
I20260620 03:06:55.753821 10545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "40154f9f64cd40228d85a33bd98028cb"
format_stamp: "Formatted at 2026-06-20 03:06:55 on dist-test-slave-xv68"
server_key: "54ab937695ef1d18f8c805e04e654d39"
server_key_iv: "9922fbe6fa676f73288e7697742bcfc2"
server_key_version: "encryptionkey@0"
I20260620 03:06:55.758564 10545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 03:06:55.761796 10559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:55.763075 10545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:06:55.763197 10545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "40154f9f64cd40228d85a33bd98028cb"
format_stamp: "Formatted at 2026-06-20 03:06:55 on dist-test-slave-xv68"
server_key: "54ab937695ef1d18f8c805e04e654d39"
server_key_iv: "9922fbe6fa676f73288e7697742bcfc2"
server_key_version: "encryptionkey@0"
I20260620 03:06:55.763460 10545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:55.768955 10545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:55.769658 10545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:55.769899 10545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:55.770705 10545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:55.772032 10545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:55.772114 10545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:55.772190 10545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:55.772244 10545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:55.805080 10545 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34249
I20260620 03:06:55.805150 10671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34249 every 8 connection(s)
I20260620 03:06:55.806377 10545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:06:55.814077 10413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:55.809009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53623 (local address 127.0.97.62:38259)
0620 03:06:55.809201 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:55.809206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:55.810566 (+ 1360us) server_negotiation.cc:407] Connection header received
0620 03:06:55.812225 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:55.812230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:55.812280 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:55.812373 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:55.813520 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:55.813524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:55.813527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:55.813542 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:55.813572 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:55.813582 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:55.813680 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:55.813839 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 03:06:55.813886 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 03:06:55.814726 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 10545
I20260620 03:06:55.814908 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:06:55.815044 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:55.809537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38259 (local address 127.0.97.3:53623)
0620 03:06:55.810419 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:55.810459 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:55.811992 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:55.812551 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:55.812559 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:55.812643 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:55.813259 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:06:55.813261 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:55.813297 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:06:55.813376 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:55.813662 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:55.813668 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:55.813674 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 03:06:55.813888 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 03:06:55.814068 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":270,"threads_started":1}
I20260620 03:06:55.815531 388 external_mini_cluster.cc:1428] Setting key 7e81b95cbfc53732d2e22fca644f6713
I20260620 03:06:55.816401 10672 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38259
I20260620 03:06:55.816720 10672 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:55.817325 10672 heartbeater.cc:511] Master 127.0.97.62:38259 requested a full tablet report, sending...
I20260620 03:06:55.818612 10225 ts_manager.cc:194] Registered new tserver with Master: 40154f9f64cd40228d85a33bd98028cb (127.0.97.3:34249)
I20260620 03:06:55.830387 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 03:06:55.836390 10413 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:06:55.835066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.0.97.62:38259)
0620 03:06:55.835390 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:55.835394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:55.835635 (+ 241us) server_negotiation.cc:407] Connection header received
0620 03:06:55.835813 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:55.835818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:55.835868 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:55.835961 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:55.836213 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41544: BlockingRecv error: recv got EOF from 127.0.0.1:41544 (error 108)
Metrics: {"server-negotiator.queue_time_us":172}
I20260620 03:06:55.837021 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 10284
I20260620 03:06:55.840310 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:55.837988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.0.97.1:42589)
0620 03:06:55.838505 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:55.838511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:55.838537 (+ 26us) server_negotiation.cc:407] Connection header received
0620 03:06:55.838597 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:55.838603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:55.838797 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:55.838921 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:55.839762 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:55.839768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:55.839780 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:55.839796 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:55.839835 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:55.839848 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:55.839954 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:55.840025 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 03:06:55.840098 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":267,"threads_started":1}
I20260620 03:06:55.841593 10406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:55.921880 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 10284
I20260620 03:06:55.938194 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 10415
I20260620 03:06:55.941922 10683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:55.939080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58084 (local address 127.0.97.2:45707)
0620 03:06:55.939674 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:55.939687 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:55.939714 (+ 27us) server_negotiation.cc:407] Connection header received
0620 03:06:55.939774 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:55.939780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:55.939992 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:55.940160 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:55.941204 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:55.941210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:55.941221 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:55.941232 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:55.941269 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:55.941282 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:55.941425 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:55.941550 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 03:06:55.941649 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":346,"threads_started":1}
I20260620 03:06:55.943174 10537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:56.025080 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 10415
I20260620 03:06:56.041283 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 10545
I20260620 03:06:56.045221 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:56.042229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.0.97.3:34249)
0620 03:06:56.042876 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:56.042883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:56.042910 (+ 27us) server_negotiation.cc:407] Connection header received
0620 03:06:56.042969 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:56.042975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:56.043185 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:56.043374 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:56.044508 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:56.044514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:56.044525 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:06:56.044547 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:56.044584 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:06:56.044597 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:56.044777 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:56.044847 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 03:06:56.044942 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":345,"threads_started":1}
I20260620 03:06:56.046597 10667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:56.128309 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 10545
I20260620 03:06:56.144580 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 10194
I20260620 03:06:56.145738 10255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:06:56.244971 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 10194
2026-06-20T03:06:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2304 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:06:56 dist-test-slave-xv68 krb5kdc[10690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:06:56 dist-test-slave-xv68 krb5kdc[10690](info): set up 2 sockets
Jun 20 03:06:56 dist-test-slave-xv68 krb5kdc[10690](info): commencing operation
krb5kdc: starting...
W20260620 03:06:57.348201 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:06:57 dist-test-slave-xv68 krb5kdc[10690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:06:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:06:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:06:57.573236 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38729
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:57.886932 10703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:57.887192 10703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:57.887399 10703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:57.894405 10703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:06:57.894495 10703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:57.894596 10703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:57.894691 10703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:06:57.894766 10703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:06:57.904258 10703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:57.905601 10703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:57.907043 10703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:57.918753 10708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:57.918833 10711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:57.918829 10709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:57.919651 10703 server_base.cc:1034] running on GCE node
I20260620 03:06:57.920398 10703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:57.921775 10703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:57.922998 10703 hybrid_clock.cc:648] HybridClock initialized: now 1781924817922956 us; error 34 us; skew 500 ppm
I20260620 03:06:57.923403 10703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:57 dist-test-slave-xv68 krb5kdc[10690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:57.932793 10703 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:06:57.934302 10703 webserver.cc:460] Webserver started at http://127.0.97.62:38859/ using document root <none> and password file <none>
I20260620 03:06:57.934985 10703 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:57.935128 10703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:57.935492 10703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:57.938349 10703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "81489c877c71436eafbae7d145b2b31e"
format_stamp: "Formatted at 2026-06-20 03:06:57 on dist-test-slave-xv68"
server_key: "fc8a589ee012efbc998bba01423e36c7"
server_key_iv: "b35e29955bd7d9bd05aeb89c8df76942"
server_key_version: "encryptionkey@0"
I20260620 03:06:57.939095 10703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "81489c877c71436eafbae7d145b2b31e"
format_stamp: "Formatted at 2026-06-20 03:06:57 on dist-test-slave-xv68"
server_key: "fc8a589ee012efbc998bba01423e36c7"
server_key_iv: "b35e29955bd7d9bd05aeb89c8df76942"
server_key_version: "encryptionkey@0"
I20260620 03:06:57.944927 10703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 03:06:57.949649 10718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:57.951249 10703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:06:57.951457 10703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "81489c877c71436eafbae7d145b2b31e"
format_stamp: "Formatted at 2026-06-20 03:06:57 on dist-test-slave-xv68"
server_key: "fc8a589ee012efbc998bba01423e36c7"
server_key_iv: "b35e29955bd7d9bd05aeb89c8df76942"
server_key_version: "encryptionkey@0"
I20260620 03:06:57.951622 10703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:57.964466 10703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:57.968078 10703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:57.968297 10703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:57.986048 10703 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38729
I20260620 03:06:57.986083 10769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38729 every 8 connection(s)
I20260620 03:06:57.987399 10703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:06:57.990904 10770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:57.993201 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 10703
I20260620 03:06:57.993477 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:06:57.994254 388 external_mini_cluster.cc:1428] Setting key d6a072b4ca38c596b3a1902b68141ced
I20260620 03:06:58.003187 10770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e: Bootstrap starting.
I20260620 03:06:58.006865 10770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:58.008162 10770 log.cc:824] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e: Log is configured to *not* fsync() on all Append() calls
Jun 20 03:06:58 dist-test-slave-xv68 krb5kdc[10690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:06:58.011163 10770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e: No bootstrap required, opened a new log
I20260620 03:06:58.017712 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:57.996271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43930 (local address 127.0.97.62:38729)
0620 03:06:57.997055 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:57.997066 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:57.997108 (+ 42us) server_negotiation.cc:407] Connection header received
0620 03:06:57.998695 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:57.998715 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:57.999073 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:57.999453 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:58.001078 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:58.002311 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:58.003999 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:58.004632 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:58.004754 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:58.009825 (+ 5071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:58.009832 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:58.009846 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:58.009884 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:58.013359 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:58.014130 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:58.014140 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:58.014147 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:58.014338 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:58.014745 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:58.014749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:58.014752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:58.015382 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:58.015744 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:58.016207 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 03:06:58.016638 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":354,"threads_started":1}
I20260620 03:06:58.019026 10770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81489c877c71436eafbae7d145b2b31e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38729 } }
I20260620 03:06:58.019400 10770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:58.019536 10770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81489c877c71436eafbae7d145b2b31e, State: Initialized, Role: FOLLOWER
I20260620 03:06:58.020077 10770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81489c877c71436eafbae7d145b2b31e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38729 } }
I20260620 03:06:58.020310 10770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:06:58.020457 10770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:06:58.020627 10770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:58.023020 10770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81489c877c71436eafbae7d145b2b31e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38729 } }
I20260620 03:06:58.023614 10770 leader_election.cc:302] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81489c877c71436eafbae7d145b2b31e; no voters:
I20260620 03:06:58.024420 10770 leader_election.cc:288] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:06:58.024574 10775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:58.025236 10775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e [term 1 LEADER]: Becoming Leader. State: Replica: 81489c877c71436eafbae7d145b2b31e, State: Running, Role: LEADER
I20260620 03:06:58.025816 10775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81489c877c71436eafbae7d145b2b31e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38729 } }
I20260620 03:06:58.026495 10770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:06:58.028867 10777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81489c877c71436eafbae7d145b2b31e. Latest consensus state: current_term: 1 leader_uuid: "81489c877c71436eafbae7d145b2b31e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81489c877c71436eafbae7d145b2b31e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38729 } } }
I20260620 03:06:58.029225 10777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:58.028854 10776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81489c877c71436eafbae7d145b2b31e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81489c877c71436eafbae7d145b2b31e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38729 } } }
I20260620 03:06:58.029508 10776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e [sys.catalog]: This master's current role is: LEADER
I20260620 03:06:58.031319 10780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:06:58.036405 10780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:06:58.044955 10780 catalog_manager.cc:1347] Generated new cluster ID: c0f84bc7b11a4580b487cd4fbcbd68c1
I20260620 03:06:58.045061 10780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:06:58.064356 10780 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:06:58.065851 10780 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:06:58.072378 10780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e: Generated new TSK 0
I20260620 03:06:58.073149 10780 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:06:58.126135 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38729
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:58.449483 10796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:58.449753 10796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:58.449846 10796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:58.457636 10796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:58.457747 10796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:58.457860 10796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:06:58.467542 10796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:58.468734 10796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:58.469957 10796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:58.482861 10801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:58.482805 10804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:58.482893 10802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:58.483781 10796 server_base.cc:1034] running on GCE node
I20260620 03:06:58.484297 10796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:58.485224 10796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:58.486441 10796 hybrid_clock.cc:648] HybridClock initialized: now 1781924818486400 us; error 28 us; skew 500 ppm
I20260620 03:06:58.486735 10796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:58 dist-test-slave-xv68 krb5kdc[10690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:58.489871 10796 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:06:58.491262 10796 webserver.cc:460] Webserver started at http://127.0.97.1:45105/ using document root <none> and password file <none>
I20260620 03:06:58.491976 10796 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:58.492118 10796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:58.492408 10796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:58.495209 10796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "350f5450fb0341f2acf6640d7339258c"
format_stamp: "Formatted at 2026-06-20 03:06:58 on dist-test-slave-xv68"
server_key: "493a97affb9ac72af802376e0e1661bf"
server_key_iv: "8262f4006f9102d756b836e1ba67c2a5"
server_key_version: "encryptionkey@0"
I20260620 03:06:58.496014 10796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "350f5450fb0341f2acf6640d7339258c"
format_stamp: "Formatted at 2026-06-20 03:06:58 on dist-test-slave-xv68"
server_key: "493a97affb9ac72af802376e0e1661bf"
server_key_iv: "8262f4006f9102d756b836e1ba67c2a5"
server_key_version: "encryptionkey@0"
I20260620 03:06:58.500970 10796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:06:58.504454 10811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:58.505800 10796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:06:58.505980 10796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "350f5450fb0341f2acf6640d7339258c"
format_stamp: "Formatted at 2026-06-20 03:06:58 on dist-test-slave-xv68"
server_key: "493a97affb9ac72af802376e0e1661bf"
server_key_iv: "8262f4006f9102d756b836e1ba67c2a5"
server_key_version: "encryptionkey@0"
I20260620 03:06:58.506167 10796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:58.518018 10796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:58.522121 10796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:58.522408 10796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:58.523324 10796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:58.524570 10796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:58.524672 10796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:58.524808 10796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:58.524876 10796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:58.558346 10796 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39411
I20260620 03:06:58.558374 10923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39411 every 8 connection(s)
I20260620 03:06:58.559695 10796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:06:58.567395 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 10796
I20260620 03:06:58.567625 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:06:58.568396 388 external_mini_cluster.cc:1428] Setting key 6310bd85d1b0ed00d2281d44243c4b95
Jun 20 03:06:58 dist-test-slave-xv68 krb5kdc[10690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:06:58.581451 10926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:58.562115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37161 (local address 127.0.97.62:38729)
0620 03:06:58.562544 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:58.562550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:58.563618 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 03:06:58.565396 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:58.565401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:58.565466 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:58.565566 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:58.566990 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:58.568163 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:58.569852 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:58.570386 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:58.570491 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:58.574622 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:58.574630 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:58.574638 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:58.574680 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:58.578094 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:58.579082 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:58.579086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:58.579088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:58.579208 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:58.579737 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:58.579741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:58.579743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:58.580012 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:58.580130 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:58.581007 (+ 877us) server_negotiation.cc:299] Negotiation successful
0620 03:06:58.581170 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":197,"threads_started":1}
I20260620 03:06:58.582922 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:58.562659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38729 (local address 127.0.97.1:37161)
0620 03:06:58.563455 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:58.563496 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:58.565109 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:58.565776 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:58.565785 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:58.566267 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:58.566799 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:58.566815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:58.568319 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:58.568322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:58.569708 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:58.569720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:58.570607 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:58.570614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:58.570901 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:58.571715 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:58.571750 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:58.574404 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:58.578275 (+ 3871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:58.578284 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:58.578301 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:58.578946 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:58.579356 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:58.579364 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:58.579370 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:58.579623 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:58.580137 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:58.580148 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:58.580583 (+ 435us) client_negotiation.cc:769] Sending connection context
0620 03:06:58.580890 (+ 307us) client_negotiation.cc:240] Negotiation successful
0620 03:06:58.581452 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":285,"threads_started":1}
I20260620 03:06:58.584506 10924 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38729
I20260620 03:06:58.584798 10924 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:58.585516 10924 heartbeater.cc:511] Master 127.0.97.62:38729 requested a full tablet report, sending...
I20260620 03:06:58.587610 10735 ts_manager.cc:194] Registered new tserver with Master: 350f5450fb0341f2acf6640d7339258c (127.0.97.1:39411)
I20260620 03:06:58.589031 10735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37161
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:06:58.620695 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38729
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:58.941208 10930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:58.941509 10930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:58.941612 10930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:58.949354 10930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:58.949473 10930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:58.949589 10930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:06:58.959420 10930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:58.960740 10930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:58.961980 10930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:58.975144 10935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:58.975247 10938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:58.975268 10936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:58.975931 10930 server_base.cc:1034] running on GCE node
I20260620 03:06:58.976465 10930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:58.977315 10930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:58.978523 10930 hybrid_clock.cc:648] HybridClock initialized: now 1781924818978474 us; error 46 us; skew 500 ppm
I20260620 03:06:58.978847 10930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:58 dist-test-slave-xv68 krb5kdc[10690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:58.982100 10930 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:06:58.983731 10930 webserver.cc:460] Webserver started at http://127.0.97.2:44709/ using document root <none> and password file <none>
I20260620 03:06:58.984426 10930 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:58.984566 10930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:58.984869 10930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:58.988638 10930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "5422921eb4354bd7ad4ce5786af2af3d"
format_stamp: "Formatted at 2026-06-20 03:06:58 on dist-test-slave-xv68"
server_key: "c4d8b75633d188e1678e7cf287b1e65d"
server_key_iv: "aa9e75fb323d381631bd007d5d57493a"
server_key_version: "encryptionkey@0"
I20260620 03:06:58.989584 10930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5422921eb4354bd7ad4ce5786af2af3d"
format_stamp: "Formatted at 2026-06-20 03:06:58 on dist-test-slave-xv68"
server_key: "c4d8b75633d188e1678e7cf287b1e65d"
server_key_iv: "aa9e75fb323d381631bd007d5d57493a"
server_key_version: "encryptionkey@0"
I20260620 03:06:58.995982 10930 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 03:06:59.000146 10945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:59.001747 10930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260620 03:06:59.001904 10930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "5422921eb4354bd7ad4ce5786af2af3d"
format_stamp: "Formatted at 2026-06-20 03:06:58 on dist-test-slave-xv68"
server_key: "c4d8b75633d188e1678e7cf287b1e65d"
server_key_iv: "aa9e75fb323d381631bd007d5d57493a"
server_key_version: "encryptionkey@0"
I20260620 03:06:59.002121 10930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:59.027531 10930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:59.031868 10930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:59.032109 10930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:59.033118 10930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:59.034485 10930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:59.034557 10930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:59.034646 10930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:59.034689 10930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:59.067961 10930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42725
I20260620 03:06:59.068058 11057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42725 every 8 connection(s)
I20260620 03:06:59.069232 10930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:06:59.072052 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 10930
I20260620 03:06:59.072263 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:06:59.072854 388 external_mini_cluster.cc:1428] Setting key eef29d7c19fba2cb4da456d8ad9bcc77
Jun 20 03:06:59 dist-test-slave-xv68 krb5kdc[10690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:06:59.091663 10926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:59.071738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58445 (local address 127.0.97.62:38729)
0620 03:06:59.071964 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:59.071970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:59.073830 (+ 1860us) server_negotiation.cc:407] Connection header received
0620 03:06:59.076063 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:59.076072 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:59.076161 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:59.076272 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:59.077774 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.078894 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.080493 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.081198 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.081354 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:59.085131 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:59.085137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:59.085140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:59.085169 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:59.088142 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:59.089211 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:59.089215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:59.089217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:59.089344 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:59.089849 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:59.089852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:59.089854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:59.090095 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:59.090198 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:59.091148 (+ 950us) server_negotiation.cc:299] Negotiation successful
0620 03:06:59.091405 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 03:06:59.093211 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:59.072344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38729 (local address 127.0.97.2:58445)
0620 03:06:59.073663 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:59.073708 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:59.075797 (+ 2089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:59.076564 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:59.076572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:59.077065 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:59.077602 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:59.077618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.079035 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.079039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:59.080322 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:59.080333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.081417 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.081422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:59.081604 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:59.082279 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:59.082301 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:59.084906 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:59.088280 (+ 3374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:59.088291 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:59.088310 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:59.089059 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:59.089482 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:59.089490 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:59.089497 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:59.089739 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:59.090228 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:59.090238 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:59.090683 (+ 445us) client_negotiation.cc:769] Sending connection context
0620 03:06:59.090996 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 03:06:59.091610 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"thread_start_us":835,"threads_started":1}
I20260620 03:06:59.094659 11058 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38729
I20260620 03:06:59.095002 11058 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:59.095917 11058 heartbeater.cc:511] Master 127.0.97.62:38729 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 03:06:59.097474 10735 ts_manager.cc:194] Registered new tserver with Master: 5422921eb4354bd7ad4ce5786af2af3d (127.0.97.2:42725)
I20260620 03:06:59.098279 10735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:06:59.126807 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38729
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:06:59.451215 11063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:06:59.451509 11063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:06:59.451679 11063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:06:59.458818 11063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:06:59.458920 11063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:06:59.459092 11063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:06:59.468776 11063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:06:59.470033 11063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:06:59.471339 11063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:06:59.484488 11068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:59.484489 11071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:06:59.484597 11069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:06:59.485646 11063 server_base.cc:1034] running on GCE node
I20260620 03:06:59.486268 11063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:06:59.487191 11063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:06:59.488451 11063 hybrid_clock.cc:648] HybridClock initialized: now 1781924819488396 us; error 84 us; skew 500 ppm
I20260620 03:06:59.488750 11063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:06:59 dist-test-slave-xv68 krb5kdc[10690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:06:59.491851 11063 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:06:59.493222 11063 webserver.cc:460] Webserver started at http://127.0.97.3:38533/ using document root <none> and password file <none>
I20260620 03:06:59.493914 11063 fs_manager.cc:359] Metadata directory not provided
I20260620 03:06:59.494052 11063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:06:59.494346 11063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:06:59.497210 11063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "bdb11c83607644c0a24c066bdb989a73"
format_stamp: "Formatted at 2026-06-20 03:06:59 on dist-test-slave-xv68"
server_key: "a656391c2a426be48bd9a2bec08114df"
server_key_iv: "54d3ea70761c8ae9b750859666a24f8a"
server_key_version: "encryptionkey@0"
I20260620 03:06:59.497972 11063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bdb11c83607644c0a24c066bdb989a73"
format_stamp: "Formatted at 2026-06-20 03:06:59 on dist-test-slave-xv68"
server_key: "a656391c2a426be48bd9a2bec08114df"
server_key_iv: "54d3ea70761c8ae9b750859666a24f8a"
server_key_version: "encryptionkey@0"
I20260620 03:06:59.502983 11063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 03:06:59.506352 11078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:59.508016 11063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 03:06:59.508180 11063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "bdb11c83607644c0a24c066bdb989a73"
format_stamp: "Formatted at 2026-06-20 03:06:59 on dist-test-slave-xv68"
server_key: "a656391c2a426be48bd9a2bec08114df"
server_key_iv: "54d3ea70761c8ae9b750859666a24f8a"
server_key_version: "encryptionkey@0"
I20260620 03:06:59.508386 11063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:06:59.526981 11063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:06:59.531322 11063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:06:59.531616 11063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:06:59.532609 11063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:06:59.534089 11063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:06:59.534188 11063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:59.534334 11063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:06:59.534410 11063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:06:59.568158 11063 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42963
I20260620 03:06:59.568209 11190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42963 every 8 connection(s)
I20260620 03:06:59.569427 11063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:06:59.577906 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 11063
I20260620 03:06:59.578120 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:06:59.578629 388 external_mini_cluster.cc:1428] Setting key 8c7c1336006841cea1f38894eaab3ef5
Jun 20 03:06:59 dist-test-slave-xv68 krb5kdc[10690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:06:59.589357 10926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:59.571959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53355 (local address 127.0.97.62:38729)
0620 03:06:59.572167 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:59.572173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:59.573270 (+ 1097us) server_negotiation.cc:407] Connection header received
0620 03:06:59.575016 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:59.575021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:59.575085 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:59.575172 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:59.576658 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.577423 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.579745 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.580255 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.580425 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:59.583987 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:59.583991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:59.583995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:59.584026 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:59.586694 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:59.587459 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:59.587463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:59.587465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:59.587595 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:59.587966 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:59.587969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:59.587971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:59.588225 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:59.588334 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:59.588967 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 03:06:59.589123 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 03:06:59.590301 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:59.572444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38729 (local address 127.0.97.3:53355)
0620 03:06:59.573108 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:59.573150 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:59.574766 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:59.575454 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:59.575462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:59.575947 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:06:59.576499 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:59.576516 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.577599 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.577606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:59.579593 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:59.579607 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.580521 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.580530 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:59.580793 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:59.581543 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:06:59.581581 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:06:59.583802 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:06:59.586816 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:59.586822 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:59.586834 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:59.587234 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:59.587687 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:06:59.587691 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:06:59.587693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:06:59.587860 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:06:59.588330 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:06:59.588336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:06:59.588646 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 03:06:59.588865 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 03:06:59.589259 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":258,"threads_started":1}
I20260620 03:06:59.591722 11191 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38729
I20260620 03:06:59.592085 11191 heartbeater.cc:464] Registering TS with master...
I20260620 03:06:59.592168 10924 heartbeater.cc:503] Master 127.0.97.62:38729 was elected leader, sending a full tablet report...
I20260620 03:06:59.592989 11191 heartbeater.cc:511] Master 127.0.97.62:38729 requested a full tablet report, sending...
I20260620 03:06:59.594458 10734 ts_manager.cc:194] Registered new tserver with Master: bdb11c83607644c0a24c066bdb989a73 (127.0.97.3:42963)
I20260620 03:06:59.595261 10734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53355
I20260620 03:06:59.606498 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:06:59.622900 10926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:59.611273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43940 (local address 127.0.97.62:38729)
0620 03:06:59.611522 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:59.611526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:59.612067 (+ 541us) server_negotiation.cc:407] Connection header received
0620 03:06:59.612201 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:59.612204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:59.612266 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:59.612348 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:06:59.613774 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.614514 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.615738 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.616124 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.616222 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:59.618239 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:06:59.618243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:06:59.618246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:06:59.618275 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:06:59.620805 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:59.621367 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:59.621371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:59.621373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:59.621486 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:06:59.621784 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:06:59.621786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:06:59.621788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:06:59.622073 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:06:59.622235 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:59.622519 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 03:06:59.622669 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260620 03:06:59.627032 10734 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:06:59.634078 10734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:06:59.668717 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:59.645530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42963 (local address 127.0.0.1:48262)
0620 03:06:59.659472 (+ 13942us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:59.659490 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:59.659674 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:59.660653 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:59.660661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:59.660710 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:59.661652 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:59.661663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.663640 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.663645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:59.665542 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:59.665556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.667545 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.667550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:59.667785 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:59.667944 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 03:06:59.668073 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 03:06:59.668263 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13827,"mutex_wait_us":397,"spinlock_wait_cycles":2816,"thread_start_us":13677,"threads_started":1}
I20260620 03:06:59.668867 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:59.645467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.0.97.1:39411)
0620 03:06:59.658349 (+ 12882us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:59.658364 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:59.659635 (+ 1271us) server_negotiation.cc:407] Connection header received
0620 03:06:59.659932 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:59.659940 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:59.660204 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:59.660387 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:59.661442 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.662898 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.665508 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.666994 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.667116 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:59.667368 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:59.668206 (+ 838us) server_negotiation.cc:299] Negotiation successful
0620 03:06:59.668423 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12630,"thread_start_us":13424,"threads_started":1}
I20260620 03:06:59.669308 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:59.660870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42725 (local address 127.0.0.1:55960)
0620 03:06:59.661758 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:59.661776 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:59.661913 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:59.662492 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:59.662497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:59.662520 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:59.662867 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:59.662886 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.664799 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.664804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:59.666413 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:59.666472 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.668578 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.668587 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:59.668769 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:59.668801 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 03:06:59.668885 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 03:06:59.669002 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":454,"threads_started":1}
I20260620 03:06:59.669346 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:59.645200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39411 (local address 127.0.0.1:54914)
0620 03:06:59.659406 (+ 14206us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:59.659455 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:59.659700 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:59.660653 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:59.660661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:59.660712 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:59.661227 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:59.661242 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.663059 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.663064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:59.665354 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:59.665365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.667532 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.667538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:59.667770 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:59.667957 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 03:06:59.668084 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 03:06:59.668876 (+ 792us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13881,"mutex_wait_us":137,"spinlock_wait_cycles":8704,"thread_start_us":13735,"threads_started":1}
I20260620 03:06:59.669648 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:59.645697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48262 (local address 127.0.97.3:42963)
0620 03:06:59.658883 (+ 13186us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:59.658893 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:59.659873 (+ 980us) server_negotiation.cc:407] Connection header received
0620 03:06:59.659956 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:59.659962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:59.660204 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:59.660387 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:59.661824 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.663400 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.665698 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.666994 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.667116 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:59.667369 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:59.669153 (+ 1784us) server_negotiation.cc:299] Negotiation successful
0620 03:06:59.669318 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13013,"thread_start_us":12866,"threads_started":1}
I20260620 03:06:59.669648 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:59.660190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55960 (local address 127.0.97.2:42725)
0620 03:06:59.661151 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:59.661161 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:59.661894 (+ 733us) server_negotiation.cc:407] Connection header received
0620 03:06:59.662061 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:59.662069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:59.662302 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:59.662484 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:59.663062 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.664655 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.666643 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.668037 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.668179 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:59.668402 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:59.669208 (+ 806us) server_negotiation.cc:299] Negotiation successful
0620 03:06:59.669353 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":734,"threads_started":1}
I20260620 03:06:59.673033 10859 tablet_service.cc:1467] Processing CreateTablet for tablet b9d73f78eda1423ab626d2f2b9a811d6 (DEFAULT_TABLE table=test-table [id=13e97ffe4c334454bc4fa09078b7d479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:59.673182 10993 tablet_service.cc:1467] Processing CreateTablet for tablet b9d73f78eda1423ab626d2f2b9a811d6 (DEFAULT_TABLE table=test-table [id=13e97ffe4c334454bc4fa09078b7d479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:59.673182 11126 tablet_service.cc:1467] Processing CreateTablet for tablet b9d73f78eda1423ab626d2f2b9a811d6 (DEFAULT_TABLE table=test-table [id=13e97ffe4c334454bc4fa09078b7d479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:06:59.674305 10859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d73f78eda1423ab626d2f2b9a811d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:59.674460 11126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d73f78eda1423ab626d2f2b9a811d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:59.674516 10993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d73f78eda1423ab626d2f2b9a811d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:06:59.685804 11209 tablet_bootstrap.cc:492] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c: Bootstrap starting.
I20260620 03:06:59.688472 11211 tablet_bootstrap.cc:492] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d: Bootstrap starting.
I20260620 03:06:59.689785 11210 tablet_bootstrap.cc:492] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73: Bootstrap starting.
I20260620 03:06:59.689994 11209 tablet_bootstrap.cc:654] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:59.691354 11209 log.cc:824] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:59.693173 11210 tablet_bootstrap.cc:654] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:59.694362 11211 tablet_bootstrap.cc:654] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d: Neither blocks nor log segments found. Creating new log.
I20260620 03:06:59.694481 11210 log.cc:824] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:59.694715 11209 tablet_bootstrap.cc:492] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c: No bootstrap required, opened a new log
I20260620 03:06:59.695003 11209 ts_tablet_manager.cc:1397] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 03:06:59.695669 11211 log.cc:824] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d: Log is configured to *not* fsync() on all Append() calls
I20260620 03:06:59.697578 11210 tablet_bootstrap.cc:492] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73: No bootstrap required, opened a new log
I20260620 03:06:59.698275 11210 ts_tablet_manager.cc:1397] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 03:06:59.698372 11211 tablet_bootstrap.cc:492] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d: No bootstrap required, opened a new log
I20260620 03:06:59.698668 11211 ts_tablet_manager.cc:1397] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260620 03:06:59.705515 11209 raft_consensus.cc:348] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350f5450fb0341f2acf6640d7339258c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39411 } } peers { permanent_uuid: "bdb11c83607644c0a24c066bdb989a73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42963 } } peers { permanent_uuid: "5422921eb4354bd7ad4ce5786af2af3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42725 } }
I20260620 03:06:59.705961 11209 raft_consensus.cc:374] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:59.706092 11209 raft_consensus.cc:729] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 350f5450fb0341f2acf6640d7339258c, State: Initialized, Role: FOLLOWER
I20260620 03:06:59.706147 11211 raft_consensus.cc:348] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350f5450fb0341f2acf6640d7339258c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39411 } } peers { permanent_uuid: "bdb11c83607644c0a24c066bdb989a73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42963 } } peers { permanent_uuid: "5422921eb4354bd7ad4ce5786af2af3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42725 } }
I20260620 03:06:59.706729 11211 raft_consensus.cc:374] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:59.706856 11211 raft_consensus.cc:729] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5422921eb4354bd7ad4ce5786af2af3d, State: Initialized, Role: FOLLOWER
I20260620 03:06:59.706766 11209 consensus_queue.cc:260] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350f5450fb0341f2acf6640d7339258c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39411 } } peers { permanent_uuid: "bdb11c83607644c0a24c066bdb989a73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42963 } } peers { permanent_uuid: "5422921eb4354bd7ad4ce5786af2af3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42725 } }
I20260620 03:06:59.707499 11211 consensus_queue.cc:260] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350f5450fb0341f2acf6640d7339258c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39411 } } peers { permanent_uuid: "bdb11c83607644c0a24c066bdb989a73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42963 } } peers { permanent_uuid: "5422921eb4354bd7ad4ce5786af2af3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42725 } }
I20260620 03:06:59.709172 11210 raft_consensus.cc:348] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350f5450fb0341f2acf6640d7339258c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39411 } } peers { permanent_uuid: "bdb11c83607644c0a24c066bdb989a73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42963 } } peers { permanent_uuid: "5422921eb4354bd7ad4ce5786af2af3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42725 } }
I20260620 03:06:59.709774 11210 raft_consensus.cc:374] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:06:59.709933 11210 raft_consensus.cc:729] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdb11c83607644c0a24c066bdb989a73, State: Initialized, Role: FOLLOWER
I20260620 03:06:59.710656 11058 heartbeater.cc:503] Master 127.0.97.62:38729 was elected leader, sending a full tablet report...
I20260620 03:06:59.711321 11211 ts_tablet_manager.cc:1428] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260620 03:06:59.712594 11210 consensus_queue.cc:260] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350f5450fb0341f2acf6640d7339258c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39411 } } peers { permanent_uuid: "bdb11c83607644c0a24c066bdb989a73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42963 } } peers { permanent_uuid: "5422921eb4354bd7ad4ce5786af2af3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42725 } }
I20260620 03:06:59.714018 11191 heartbeater.cc:503] Master 127.0.97.62:38729 was elected leader, sending a full tablet report...
I20260620 03:06:59.714617 11210 ts_tablet_manager.cc:1428] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260620 03:06:59.716092 11209 ts_tablet_manager.cc:1428] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c: Time spent starting tablet: real 0.021s user 0.018s sys 0.000s
W20260620 03:06:59.812317 10925 tablet.cc:2369] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:06:59.821071 11192 tablet.cc:2369] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:06:59.821115 11059 tablet.cc:2369] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:06:59.970152 11217 raft_consensus.cc:482] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:06:59.970435 11217 raft_consensus.cc:504] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350f5450fb0341f2acf6640d7339258c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39411 } } peers { permanent_uuid: "bdb11c83607644c0a24c066bdb989a73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42963 } } peers { permanent_uuid: "5422921eb4354bd7ad4ce5786af2af3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42725 } }
I20260620 03:06:59.972572 11217 leader_election.cc:288] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 350f5450fb0341f2acf6640d7339258c (127.0.97.1:39411), 5422921eb4354bd7ad4ce5786af2af3d (127.0.97.2:42725)
I20260620 03:06:59.978550 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:59.973150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50873 (local address 127.0.97.2:42725)
0620 03:06:59.973370 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:59.973376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:59.973397 (+ 21us) server_negotiation.cc:407] Connection header received
0620 03:06:59.973474 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:59.973485 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:59.973565 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:59.973706 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:59.974142 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.974922 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.976719 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.977567 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.977723 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:59.977819 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:59.978125 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 03:06:59.978256 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:06:59.978577 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:59.972836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42725 (local address 127.0.97.3:50873)
0620 03:06:59.972994 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:59.973013 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:59.973138 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:59.973743 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:59.973748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:59.973778 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:59.973983 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:59.973992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.975067 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.975075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:59.976542 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:59.976554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.977736 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.977742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:59.977939 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:59.977964 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 03:06:59.978100 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 03:06:59.978249 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 03:06:59.978736 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:59.973027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51083 (local address 127.0.97.1:39411)
0620 03:06:59.973234 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 03:06:59.973241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:06:59.973851 (+ 610us) server_negotiation.cc:407] Connection header received
0620 03:06:59.974052 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:06:59.974056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:06:59.974113 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:06:59.974205 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:06:59.974681 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.975430 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.977060 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.977838 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.977981 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:59.978086 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 03:06:59.978370 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 03:06:59.978491 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 03:06:59.978901 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:06:59.972842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39411 (local address 127.0.97.3:51083)
0620 03:06:59.973746 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0620 03:06:59.973760 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 03:06:59.973914 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:06:59.974228 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:06:59.974233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:06:59.974255 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:06:59.974535 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:59.974542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.975565 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.975571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:59.976909 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:06:59.976919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:06:59.978076 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:06:59.978082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:06:59.978218 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:06:59.978239 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 03:06:59.978339 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 03:06:59.978453 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":135,"thread_start_us":462,"threads_started":1}
I20260620 03:06:59.979712 11013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d73f78eda1423ab626d2f2b9a811d6" candidate_uuid: "bdb11c83607644c0a24c066bdb989a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5422921eb4354bd7ad4ce5786af2af3d" is_pre_election: true
I20260620 03:06:59.979822 10879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d73f78eda1423ab626d2f2b9a811d6" candidate_uuid: "bdb11c83607644c0a24c066bdb989a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "350f5450fb0341f2acf6640d7339258c" is_pre_election: true
I20260620 03:06:59.980221 11013 raft_consensus.cc:2454] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdb11c83607644c0a24c066bdb989a73 in term 0.
I20260620 03:06:59.980399 10879 raft_consensus.cc:2454] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdb11c83607644c0a24c066bdb989a73 in term 0.
I20260620 03:06:59.980890 11079 leader_election.cc:302] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5422921eb4354bd7ad4ce5786af2af3d, bdb11c83607644c0a24c066bdb989a73; no voters:
I20260620 03:06:59.981287 11217 raft_consensus.cc:2781] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:06:59.981446 11217 raft_consensus.cc:482] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:06:59.981572 11217 raft_consensus.cc:3037] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:59.984105 11217 raft_consensus.cc:504] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350f5450fb0341f2acf6640d7339258c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39411 } } peers { permanent_uuid: "bdb11c83607644c0a24c066bdb989a73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42963 } } peers { permanent_uuid: "5422921eb4354bd7ad4ce5786af2af3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42725 } }
I20260620 03:06:59.984831 11217 leader_election.cc:288] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [CANDIDATE]: Term 1 election: Requested vote from peers 350f5450fb0341f2acf6640d7339258c (127.0.97.1:39411), 5422921eb4354bd7ad4ce5786af2af3d (127.0.97.2:42725)
I20260620 03:06:59.985319 11013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d73f78eda1423ab626d2f2b9a811d6" candidate_uuid: "bdb11c83607644c0a24c066bdb989a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5422921eb4354bd7ad4ce5786af2af3d"
I20260620 03:06:59.985435 10879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d73f78eda1423ab626d2f2b9a811d6" candidate_uuid: "bdb11c83607644c0a24c066bdb989a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "350f5450fb0341f2acf6640d7339258c"
I20260620 03:06:59.985526 11013 raft_consensus.cc:3037] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:59.985626 10879 raft_consensus.cc:3037] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:06:59.988704 10879 raft_consensus.cc:2454] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdb11c83607644c0a24c066bdb989a73 in term 1.
I20260620 03:06:59.988704 11013 raft_consensus.cc:2454] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdb11c83607644c0a24c066bdb989a73 in term 1.
I20260620 03:06:59.989113 11081 leader_election.cc:302] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 350f5450fb0341f2acf6640d7339258c, bdb11c83607644c0a24c066bdb989a73; no voters:
I20260620 03:06:59.989414 11217 raft_consensus.cc:2781] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:06:59.989880 11217 raft_consensus.cc:686] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [term 1 LEADER]: Becoming Leader. State: Replica: bdb11c83607644c0a24c066bdb989a73, State: Running, Role: LEADER
I20260620 03:06:59.990393 11217 consensus_queue.cc:237] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350f5450fb0341f2acf6640d7339258c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39411 } } peers { permanent_uuid: "bdb11c83607644c0a24c066bdb989a73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42963 } } peers { permanent_uuid: "5422921eb4354bd7ad4ce5786af2af3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42725 } }
I20260620 03:06:59.994597 10734 catalog_manager.cc:5495] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdb11c83607644c0a24c066bdb989a73 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "bdb11c83607644c0a24c066bdb989a73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350f5450fb0341f2acf6640d7339258c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdb11c83607644c0a24c066bdb989a73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5422921eb4354bd7ad4ce5786af2af3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42725 } health_report { overall_health: UNKNOWN } } }
I20260620 03:07:00.036032 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:00.030546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.0.97.3:42963)
0620 03:07:00.030766 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 03:07:00.030773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:07:00.030793 (+ 20us) server_negotiation.cc:407] Connection header received
0620 03:07:00.031032 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:07:00.031036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:07:00.031120 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:07:00.031279 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:07:00.031669 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:00.032466 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:00.033969 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:00.034348 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:00.034453 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:00.034901 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:07:00.035496 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:07:00.035610 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 03:07:00.035761 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 03:07:00.035825 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 03:07:00.043893 10879 raft_consensus.cc:1261] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c [term 1 FOLLOWER]: Refusing update from remote peer bdb11c83607644c0a24c066bdb989a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:07:00.043907 11013 raft_consensus.cc:1261] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d [term 1 FOLLOWER]: Refusing update from remote peer bdb11c83607644c0a24c066bdb989a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:07:00.045051 11219 consensus_queue.cc:1035] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "350f5450fb0341f2acf6640d7339258c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:07:00.045426 11217 consensus_queue.cc:1035] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5422921eb4354bd7ad4ce5786af2af3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:07:00.056231 11223 mvcc.cc:205] Tried to move back new op lower bound from 7298764062889771008 to 7298764062684434432. Current Snapshot: MvccSnapshot[applied={T|T < 7298764062889771008}]
I20260620 03:07:00.058979 11222 mvcc.cc:205] Tried to move back new op lower bound from 7298764062889771008 to 7298764062684434432. Current Snapshot: MvccSnapshot[applied={T|T < 7298764062889771008}]
I20260620 03:07:00.072460 10735 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:07:00.072724 10735 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:07:00.076467 10735 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81489c877c71436eafbae7d145b2b31e: Sending DeleteTablet for 3 replicas of tablet b9d73f78eda1423ab626d2f2b9a811d6
I20260620 03:07:00.077811 10993 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d73f78eda1423ab626d2f2b9a811d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:07:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55960
I20260620 03:07:00.078927 10859 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d73f78eda1423ab626d2f2b9a811d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:07:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54914
I20260620 03:07:00.079017 11232 tablet_replica.cc:330] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d: stopping tablet replica
I20260620 03:07:00.079443 11232 raft_consensus.cc:2229] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:07:00.080036 11232 raft_consensus.cc:2258] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:07:00.080044 11126 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d73f78eda1423ab626d2f2b9a811d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:07:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48262
I20260620 03:07:00.082197 11232 ts_tablet_manager.cc:1905] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:07:00.085103 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 10796
I20260620 03:07:00.087729 11233 tablet_replica.cc:330] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c: stopping tablet replica
I20260620 03:07:00.088162 11233 raft_consensus.cc:2229] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:07:00.088668 11233 raft_consensus.cc:2258] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:07:00.089524 11232 ts_tablet_manager.cc:1918] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:07:00.089736 11232 log.cc:1196] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/b9d73f78eda1423ab626d2f2b9a811d6
I20260620 03:07:00.090314 11232 ts_tablet_manager.cc:1939] T b9d73f78eda1423ab626d2f2b9a811d6 P 5422921eb4354bd7ad4ce5786af2af3d: Deleting consensus metadata
I20260620 03:07:00.090798 11233 ts_tablet_manager.cc:1905] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:07:00.092360 10719 catalog_manager.cc:4841] TS 5422921eb4354bd7ad4ce5786af2af3d (127.0.97.2:42725): tablet b9d73f78eda1423ab626d2f2b9a811d6 (table test-table [id=13e97ffe4c334454bc4fa09078b7d479]) successfully deleted
I20260620 03:07:00.093670 11234 tablet_replica.cc:330] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73: stopping tablet replica
I20260620 03:07:00.094081 11234 raft_consensus.cc:2229] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:07:00.094695 11234 raft_consensus.cc:2258] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:07:00.096863 11234 ts_tablet_manager.cc:1905] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:07:00.097889 11233 ts_tablet_manager.cc:1918] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:07:00.098105 11233 log.cc:1196] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/b9d73f78eda1423ab626d2f2b9a811d6
I20260620 03:07:00.098706 11233 ts_tablet_manager.cc:1939] T b9d73f78eda1423ab626d2f2b9a811d6 P 350f5450fb0341f2acf6640d7339258c: Deleting consensus metadata
I20260620 03:07:00.100443 10721 catalog_manager.cc:4841] TS 350f5450fb0341f2acf6640d7339258c (127.0.97.1:39411): tablet b9d73f78eda1423ab626d2f2b9a811d6 (table test-table [id=13e97ffe4c334454bc4fa09078b7d479]) successfully deleted
Jun 20 03:07:00 dist-test-slave-xv68 krb5kdc[10690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:07:00.103338 11234 ts_tablet_manager.cc:1918] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:07:00.103508 11234 log.cc:1196] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/b9d73f78eda1423ab626d2f2b9a811d6
I20260620 03:07:00.104063 11234 ts_tablet_manager.cc:1939] T b9d73f78eda1423ab626d2f2b9a811d6 P bdb11c83607644c0a24c066bdb989a73: Deleting consensus metadata
I20260620 03:07:00.105557 10719 catalog_manager.cc:4841] TS bdb11c83607644c0a24c066bdb989a73 (127.0.97.3:42963): tablet b9d73f78eda1423ab626d2f2b9a811d6 (table test-table [id=13e97ffe4c334454bc4fa09078b7d479]) successfully deleted
I20260620 03:07:00.110996 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:00.086459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54926 (local address 127.0.97.1:39411)
0620 03:07:00.086611 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 03:07:00.086617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:07:00.092152 (+ 5535us) server_negotiation.cc:407] Connection header received
0620 03:07:00.092694 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:07:00.092705 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:07:00.092804 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:07:00.093045 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:07:00.095009 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:00.096255 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:00.098079 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:00.098745 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:00.098839 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:00.104112 (+ 5273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:07:00.104129 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:07:00.104147 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:07:00.104191 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:07:00.108007 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:00.108739 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:00.108749 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:00.108753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:00.108905 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:00.109312 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:00.109317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:00.109320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:00.109749 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:07:00.110053 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 03:07:00.110526 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 03:07:00.110691 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 03:07:00.112354 10919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:07:00.221493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 10796
I20260620 03:07:00.238946 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 10930
Jun 20 03:07:00 dist-test-slave-xv68 krb5kdc[10690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:07:00.255625 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:00.239805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55962 (local address 127.0.97.2:42725)
0620 03:07:00.239999 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 03:07:00.240006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:07:00.240027 (+ 21us) server_negotiation.cc:407] Connection header received
0620 03:07:00.240152 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:07:00.240158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:07:00.240246 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:07:00.240359 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:07:00.241541 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:00.242621 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:00.243985 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:00.244611 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:00.244741 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:00.249247 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:07:00.249265 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:07:00.249287 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:07:00.249327 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:07:00.252813 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:00.253422 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:00.253431 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:00.253435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:00.253589 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:00.253915 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:00.253920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:00.253923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:00.254370 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:07:00.254667 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 03:07:00.254990 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 03:07:00.255193 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 03:07:00.256417 11053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:07:00.344682 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 10930
I20260620 03:07:00.362938 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 11063
Jun 20 03:07:00 dist-test-slave-xv68 krb5kdc[10690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:07:00.378768 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:00.363744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.0.97.3:42963)
0620 03:07:00.363942 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 03:07:00.363948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:07:00.363972 (+ 24us) server_negotiation.cc:407] Connection header received
0620 03:07:00.364143 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:07:00.364151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:07:00.364225 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:07:00.364367 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:07:00.365557 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:00.366342 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:00.367800 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:00.368240 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:00.368389 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:00.372594 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:07:00.372606 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:07:00.372621 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:07:00.372659 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:07:00.375987 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:00.376732 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:00.376742 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:00.376745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:00.376896 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:00.377267 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:00.377273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:00.377276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:00.377609 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:07:00.377886 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 03:07:00.378184 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 03:07:00.378407 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 03:07:00.379491 11186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:07:00.498926 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 11063
I20260620 03:07:00.520318 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 10703
I20260620 03:07:00.521377 10765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:07:00.626652 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 10703
2026-06-20T03:07:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4383 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:07:00 dist-test-slave-xv68 krb5kdc[11242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:07:00 dist-test-slave-xv68 krb5kdc[11242](info): set up 2 sockets
Jun 20 03:07:00 dist-test-slave-xv68 krb5kdc[11242](info): commencing operation
krb5kdc: starting...
W20260620 03:07:01.736150 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:07:01 dist-test-slave-xv68 krb5kdc[11242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:07:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:07:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 03:07:02.038090 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5VLrcV/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/postgres -l logfile start
2026-06-20 03:07:04.501 UTC [11272] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 03:07:04.501 UTC [11272] LOG: listening on IPv4 address "127.0.97.20", port 47917
2026-06-20 03:07:04.505 UTC [11272] LOG: listening on Unix socket "/tmp/.s.PGSQL.47917"
2026-06-20 03:07:04.527 UTC [11275] LOG: database system was shut down at 2026-06-20 03:07:03 UTC
2026-06-20 03:07:04.530 UTC [11272] LOG: database system is ready to accept connections
I20260620 03:07:05.507818 388 mini_postgres.cc:96] Postgres bound to 47917
2026-06-20 03:07:05.525 UTC [11283] FATAL: database "slave" does not exist
127.0.97.20:47917 - accepting connections
I20260620 03:07:05.525882 388 mini_ranger.cc:161] Starting Ranger...
I20260620 03:07:05.551090 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 03:07:06.051599 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 03:07:06.051718 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ranger-admin
2026-06-20 03:07:06,334 [I] DB FLAVOR :POSTGRES
2026-06-20 03:07:06,334 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 03:07:06,334 [I] Checking connection..
2026-06-20 03:07:06,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 03:07:06,570 [I] Checking connection passed.
2026-06-20 03:07:06,570 [I] --------- Verifying version history table ---------
2026-06-20 03:07:06,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 03:07:06,826 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 03:07:06,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 03:07:07,079 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 03:07:07,079 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 03:07:07,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 03:07:07.322 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 03:07:07,328 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 03:07:07,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 03:07:07,570 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 03:07:07,571 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 03:07:07,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 03:07:07,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 03:07:08,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-xv68.c.gcp-upstream.internal','N') ;"
2026-06-20 03:07:08,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 03:07:08,515 [I] Table x_portal_user does not exist in database ranger
2026-06-20 03:07:08,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 03:07:08,769 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 03:07:08,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 03:07:08,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 03:07:09,233 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 03:07:09,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 03:07:10.161 UTC [11455] WARNING: there is no transaction in progress
2026-06-20 03:07:10.177 UTC [11455] WARNING: there is no transaction in progress
2026-06-20 03:07:10.193 UTC [11455] WARNING: there is no transaction in progress
2026-06-20 03:07:10.207 UTC [11455] WARNING: there is no transaction in progress
2026-06-20 03:07:10.223 UTC [11455] WARNING: there is no transaction in progress
2026-06-20 03:07:10.243 UTC [11455] WARNING: there is no transaction in progress
2026-06-20 03:07:10.305 UTC [11455] WARNING: there is no transaction in progress
2026-06-20 03:07:10.312 UTC [11455] WARNING: there is no transaction in progress
2026-06-20 03:07:10.326 UTC [11455] WARNING: there is no transaction in progress
2026-06-20 03:07:10.335 UTC [11455] WARNING: there is no transaction in progress
2026-06-20 03:07:10.345 UTC [11455] WARNING: there is no transaction in progress
2026-06-20 03:07:10.358 UTC [11455] WARNING: there is no transaction in progress
2026-06-20 03:07:10.367 UTC [11455] WARNING: there is no transaction in progress
2026-06-20 03:07:10.373 UTC [11455] WARNING: there is no transaction in progress
2026-06-20 03:07:10.380 UTC [11455] WARNING: there is no transaction in progress
2026-06-20 03:07:10.386 UTC [11455] WARNING: there is no transaction in progress
2026-06-20 03:07:11,000 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 03:07:11,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 03:07:11,254 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 03:07:11,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 03:07:11,504 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 03:07:11,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 03:07:11,761 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 03:07:11,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 03:07:11,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 03:07:12,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-xv68.c.gcp-upstream.internal';"
2026-06-20 03:07:12,444 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 03:07:12,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 03:07:12,674 [I] DB_PATCHES have already been applied
I20260620 03:07:12.680330 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5VLrcV/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 03:07:12.680449 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 03:07:12.695171 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:52405
Jun 20, 2026 3:07:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:07:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:07:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 3:07:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 3:07:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 3:07:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 3:07:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 03:07:13.841393 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.316s user 0.005s sys 0.034s
Jun 20, 2026 3:07:33 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 3:07:33 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 3:07:33 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 3:07:33 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 3:07:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 3:07:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 3:07:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 3:07:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 3:07:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 3:07:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 3:07:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 3:07:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 3:07:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 3:07:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 3:07:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 3:07:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 3:07:33 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 3:07:34 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 03:07:34.850147 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 03:07:35 dist-test-slave-xv68 krb5kdc[11242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924855, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 03:07:35.014252 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 03:07:35.033311 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 03:07:35.913146 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 03:07:35.913318 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ranger-kms
I20260620 03:07:35.913455 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:47917
2026-06-20 03:07:36,357 [I] DB FLAVOR :POSTGRES
2026-06-20 03:07:36,357 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 03:07:36,357 [I] Checking connection
2026-06-20 03:07:36,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 03:07:36,605 [I] connection success
2026-06-20 03:07:36,605 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 03:07:36,606 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 03:07:36,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 03:07:36,904 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 03:07:36,904 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 03:07:36,904 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 03:07:36,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47917/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 03:07:37,160 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 03:07:37.649641 388 mini_ranger_kms.cc:310] Created kms service
I20260620 03:07:37.857568 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 03:07:37.900835 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 03:07:37 dist-test-slave-xv68 krb5kdc[11242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924855, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:07:38.267696 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 03:07:38.268009 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 03:07:38.268049 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 03:07:38.286228 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11635
I20260620 03:07:38.286437 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:49057
Jun 20, 2026 3:07:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:07:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:07:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 3:07:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 3:07:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 3:07:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 03:07:40 dist-test-slave-xv68 krb5kdc[11242](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781924860, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:07:40.582060 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.568s user 0.132s sys 0.325s
I20260620 03:07:40.582315 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 03:07:40.582535 388 mini_ranger_kms.cc:401] 127.0.97.20:49057/kms/v1/keys
Jun 20 03:07:40 dist-test-slave-xv68 krb5kdc[11242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924860, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 03:07:41 dist-test-slave-xv68 krb5kdc[11242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924860, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 03:07:43 dist-test-slave-xv68 krb5kdc[11242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:07:43.514665 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38503
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38503
--ranger_config_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49057/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:07:43.851006 11686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:07:43.851372 11686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:07:43.851531 11686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:07:43.858913 11686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:07:43.859050 11686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:07:43.859155 11686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:07:43.859261 11686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:07:43.859373 11686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:07:43.869092 11686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:49057/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38503
--ranger_config_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:07:43.870292 11686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:07:43.871553 11686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:07:43.885036 11691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:07:43.885111 11692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:07:43.885070 11694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:07:43.885393 11686 server_base.cc:1034] running on GCE node
I20260620 03:07:43.886204 11686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:07:43.887768 11686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:07:43.889001 11686 hybrid_clock.cc:648] HybridClock initialized: now 1781924863888976 us; error 56 us; skew 500 ppm
I20260620 03:07:43.889355 11686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:07:43 dist-test-slave-xv68 krb5kdc[11242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:07:43.895516 11686 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:07:43.897053 11686 webserver.cc:460] Webserver started at http://127.0.97.62:37769/ using document root <none> and password file <none>
I20260620 03:07:43.897707 11686 fs_manager.cc:359] Metadata directory not provided
I20260620 03:07:43.897847 11686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:07:43.898142 11686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 03:07:43 dist-test-slave-xv68 krb5kdc[11242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:07:43.950461 11686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "71ee6b246d78456dbfa7242f6360dd2f"
format_stamp: "Formatted at 2026-06-20 03:07:43 on dist-test-slave-xv68"
server_key: "8af55503f406c02fec83e8c20e3f6111"
server_key_iv: "bbb82c97c672dadf9503dd16ac496fb3"
server_key_version: "kuduclusterkey@0"
I20260620 03:07:43.951404 11686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "71ee6b246d78456dbfa7242f6360dd2f"
format_stamp: "Formatted at 2026-06-20 03:07:43 on dist-test-slave-xv68"
server_key: "8af55503f406c02fec83e8c20e3f6111"
server_key_iv: "bbb82c97c672dadf9503dd16ac496fb3"
server_key_version: "kuduclusterkey@0"
I20260620 03:07:43.957026 11686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 03:07:43.973407 11701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:07:43.975049 11686 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 03:07:43.975231 11686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "71ee6b246d78456dbfa7242f6360dd2f"
format_stamp: "Formatted at 2026-06-20 03:07:43 on dist-test-slave-xv68"
server_key: "8af55503f406c02fec83e8c20e3f6111"
server_key_iv: "bbb82c97c672dadf9503dd16ac496fb3"
server_key_version: "kuduclusterkey@0"
I20260620 03:07:43.975518 11686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:07:44.012841 11686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:07:44.017095 11686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:07:44.017414 11686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:07:44.037804 11686 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38503
I20260620 03:07:44.037822 11752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38503 every 8 connection(s)
I20260620 03:07:44.039151 11686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:07:44.042953 11753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:07:44.043406 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 11686
I20260620 03:07:44.043598 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:07:44.044450 388 external_mini_cluster.cc:1428] Setting key a0df7f29de2cea05c6a9c2e824154b3b
I20260620 03:07:44.061684 11753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f: Bootstrap starting.
I20260620 03:07:44.065475 11753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f: Neither blocks nor log segments found. Creating new log.
I20260620 03:07:44.066795 11753 log.cc:824] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f: Log is configured to *not* fsync() on all Append() calls
I20260620 03:07:44.070431 11753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f: No bootstrap required, opened a new log
Jun 20 03:07:44 dist-test-slave-xv68 krb5kdc[11242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:07:44.080714 11753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ee6b246d78456dbfa7242f6360dd2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38503 } }
I20260620 03:07:44.081133 11753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:07:44.081287 11753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ee6b246d78456dbfa7242f6360dd2f, State: Initialized, Role: FOLLOWER
I20260620 03:07:44.081929 11753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ee6b246d78456dbfa7242f6360dd2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38503 } }
I20260620 03:07:44.082154 11753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:07:44.082280 11753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:07:44.082480 11753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:07:44.084481 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:44.046586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.0.97.62:38503)
0620 03:07:44.059917 (+ 13331us) server_negotiation.cc:206] Beginning negotiation
0620 03:07:44.059937 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 03:07:44.060370 (+ 433us) server_negotiation.cc:407] Connection header received
0620 03:07:44.062717 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:07:44.062743 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:07:44.063225 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:07:44.063734 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:07:44.065155 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:44.067231 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:44.068702 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:44.069704 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:44.069901 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:44.074612 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:07:44.074623 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:07:44.074642 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:07:44.074688 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:07:44.078715 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:44.079554 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:44.079561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:44.079571 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:44.079749 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:44.080338 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:44.080342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:44.080346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:44.081074 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:07:44.081364 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 03:07:44.082621 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0620 03:07:44.083042 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13062,"thread_start_us":12827,"threads_started":1}
I20260620 03:07:44.084973 11753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ee6b246d78456dbfa7242f6360dd2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38503 } }
I20260620 03:07:44.085944 11753 leader_election.cc:302] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71ee6b246d78456dbfa7242f6360dd2f; no voters:
I20260620 03:07:44.088143 11753 leader_election.cc:288] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:07:44.088253 11758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:07:44.090207 11753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:07:44.095176 11758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f [term 1 LEADER]: Becoming Leader. State: Replica: 71ee6b246d78456dbfa7242f6360dd2f, State: Running, Role: LEADER
I20260620 03:07:44.095866 11758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ee6b246d78456dbfa7242f6360dd2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38503 } }
I20260620 03:07:44.099498 11759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71ee6b246d78456dbfa7242f6360dd2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ee6b246d78456dbfa7242f6360dd2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38503 } } }
I20260620 03:07:44.099864 11760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71ee6b246d78456dbfa7242f6360dd2f. Latest consensus state: current_term: 1 leader_uuid: "71ee6b246d78456dbfa7242f6360dd2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ee6b246d78456dbfa7242f6360dd2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38503 } } }
I20260620 03:07:44.100307 11759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f [sys.catalog]: This master's current role is: LEADER
I20260620 03:07:44.100309 11760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f [sys.catalog]: This master's current role is: LEADER
I20260620 03:07:44.108541 11768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:07:44.113801 11768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:07:44.114537 11753 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 03:07:44.147400 11768 catalog_manager.cc:1347] Generated new cluster ID: 13887daae925465998ea37a390b03335
I20260620 03:07:44.147720 11768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:07:44.184835 11768 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:07:44.186998 11768 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:07:44.197011 11768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f: Generated new TSK 0
I20260620 03:07:44.198117 11768 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:07:44.911998 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49057/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38503
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:07:45 dist-test-slave-xv68 krb5kdc[11242](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781924865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 03:07:45.287341 11798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:07:45.287669 11798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:07:45.287845 11798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:07:45.296003 11798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:07:45.296139 11798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:07:45.296320 11798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:07:45.307602 11798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:49057/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:07:45.309168 11798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:07:45.310454 11798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:07:45.327981 11808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:07:45.328553 11809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:07:45.333814 11798 server_base.cc:1034] running on GCE node
W20260620 03:07:45.334133 11811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:07:45.334834 11798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:07:45.335955 11798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:07:45.337218 11798 hybrid_clock.cc:648] HybridClock initialized: now 1781924865337153 us; error 60 us; skew 500 ppm
I20260620 03:07:45.337550 11798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:07:45 dist-test-slave-xv68 krb5kdc[11242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:07:45.341665 11798 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:07:45.345513 11798 webserver.cc:460] Webserver started at http://127.0.97.1:34427/ using document root <none> and password file <none>
I20260620 03:07:45.346313 11798 fs_manager.cc:359] Metadata directory not provided
I20260620 03:07:45.346469 11798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:07:45.346797 11798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 03:07:45 dist-test-slave-xv68 krb5kdc[11242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:07:45.364445 11798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "d08c040bc05b4a9d92f679bb8941184a"
format_stamp: "Formatted at 2026-06-20 03:07:45 on dist-test-slave-xv68"
server_key: "bde6a23d5b1f23fa66c094981cb3bfb3"
server_key_iv: "db1ff13eafdedfb83a6d02a5295bc6ba"
server_key_version: "kuduclusterkey@0"
I20260620 03:07:45.365729 11798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d08c040bc05b4a9d92f679bb8941184a"
format_stamp: "Formatted at 2026-06-20 03:07:45 on dist-test-slave-xv68"
server_key: "bde6a23d5b1f23fa66c094981cb3bfb3"
server_key_iv: "db1ff13eafdedfb83a6d02a5295bc6ba"
server_key_version: "kuduclusterkey@0"
I20260620 03:07:45.373107 11798 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.006s
I20260620 03:07:45.402024 11818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:07:45.404644 11798 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 03:07:45.404862 11798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "d08c040bc05b4a9d92f679bb8941184a"
format_stamp: "Formatted at 2026-06-20 03:07:45 on dist-test-slave-xv68"
server_key: "bde6a23d5b1f23fa66c094981cb3bfb3"
server_key_iv: "db1ff13eafdedfb83a6d02a5295bc6ba"
server_key_version: "kuduclusterkey@0"
I20260620 03:07:45.405164 11798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:07:45.446856 11798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:07:45.452322 11798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:07:45.452680 11798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:07:45.453964 11798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:07:45.456245 11798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:07:45.456346 11798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:07:45.456517 11798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:07:45.456584 11798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:07:45.546806 11798 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44389
I20260620 03:07:45.546829 11930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44389 every 8 connection(s)
I20260620 03:07:45.548143 11798 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:07:45.548770 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 11798
I20260620 03:07:45.548969 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:07:45.549772 388 external_mini_cluster.cc:1428] Setting key 97cc8817713509d04ceabeb236999599
Jun 20 03:07:45 dist-test-slave-xv68 krb5kdc[11242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 03:07:45.592792 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:45.552735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34239 (local address 127.0.97.62:38503)
0620 03:07:45.567707 (+ 14972us) server_negotiation.cc:206] Beginning negotiation
0620 03:07:45.567714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:07:45.571355 (+ 3641us) server_negotiation.cc:407] Connection header received
0620 03:07:45.571483 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:07:45.571489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:07:45.571590 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:07:45.571679 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:07:45.573940 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:45.575100 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:45.577527 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:45.578162 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:45.578279 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:45.584407 (+ 6128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:07:45.584414 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:07:45.584420 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:07:45.584475 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:07:45.588648 (+ 4173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:45.589658 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:45.589666 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:45.589671 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:45.589821 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:45.590504 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:45.590512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:45.590518 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:45.590930 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:07:45.591063 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 03:07:45.592160 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0620 03:07:45.592420 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14823,"thread_start_us":14684,"threads_started":1}
I20260620 03:07:45.594125 11934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:45.553548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38503 (local address 127.0.97.1:34239)
0620 03:07:45.568576 (+ 15028us) negotiation.cc:107] Waiting for socket to connect
0620 03:07:45.568634 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 03:07:45.571059 (+ 2425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:07:45.572254 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:07:45.572280 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:07:45.572919 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:07:45.573734 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:07:45.573759 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:07:45.575336 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:07:45.575342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:07:45.577361 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:07:45.577376 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:07:45.578628 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:07:45.578638 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:07:45.578945 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:45.581302 (+ 2357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:07:45.581335 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:07:45.584105 (+ 2770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:07:45.588837 (+ 4732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:07:45.588847 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:07:45.588867 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:07:45.589503 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:07:45.589976 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:07:45.589981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:07:45.589986 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:07:45.590358 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:07:45.591151 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:07:45.591158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:07:45.591794 (+ 636us) client_negotiation.cc:769] Sending connection context
0620 03:07:45.592070 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 03:07:45.592613 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14711,"thread_start_us":13603,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 03:07:45.599392 11931 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38503
I20260620 03:07:45.599722 11931 heartbeater.cc:464] Registering TS with master...
I20260620 03:07:45.603461 11931 heartbeater.cc:511] Master 127.0.97.62:38503 requested a full tablet report, sending...
I20260620 03:07:45.606235 11718 ts_manager.cc:194] Registered new tserver with Master: d08c040bc05b4a9d92f679bb8941184a (127.0.97.1:44389)
I20260620 03:07:45.608664 11718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:07:45.640118 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49057/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38503
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:07:45 dist-test-slave-xv68 krb5kdc[11242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 03:07:46 dist-test-slave-xv68 krb5kdc[11242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 03:07:46.116667 11937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:07:46.117198 11937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:07:46.117409 11937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:07:46.127498 11937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:07:46.127646 11937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:07:46.127874 11937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:07:46.143592 11937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:49057/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:07:46.145530 11937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:07:46.147390 11937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:07:46.181459 11945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:07:46.185690 11944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:07:46.188445 11948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:07:46.188879 11937 server_base.cc:1034] running on GCE node
I20260620 03:07:46.190027 11937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:07:46.195652 11937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:07:46.196993 11937 hybrid_clock.cc:648] HybridClock initialized: now 1781924866196913 us; error 78 us; skew 500 ppm
I20260620 03:07:46.197378 11937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:07:46 dist-test-slave-xv68 krb5kdc[11242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:07:46.201385 11937 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:07:46.239818 11937 webserver.cc:460] Webserver started at http://127.0.97.2:46737/ using document root <none> and password file <none>
I20260620 03:07:46.241134 11937 fs_manager.cc:359] Metadata directory not provided
I20260620 03:07:46.241379 11937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:07:46.241816 11937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 03:07:46 dist-test-slave-xv68 krb5kdc[11242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:07:46.263779 11937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "235672e6f02f449f9d49b944afc8e37d"
format_stamp: "Formatted at 2026-06-20 03:07:46 on dist-test-slave-xv68"
server_key: "b3fb6c149263a22d949504948cff1b5b"
server_key_iv: "fd7ef00365fd85bba3e8379af64b376a"
server_key_version: "kuduclusterkey@0"
I20260620 03:07:46.265029 11937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "235672e6f02f449f9d49b944afc8e37d"
format_stamp: "Formatted at 2026-06-20 03:07:46 on dist-test-slave-xv68"
server_key: "b3fb6c149263a22d949504948cff1b5b"
server_key_iv: "fd7ef00365fd85bba3e8379af64b376a"
server_key_version: "kuduclusterkey@0"
I20260620 03:07:46.279073 11937 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.007s sys 0.004s
I20260620 03:07:46.328001 11954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:07:46.339445 11937 fs_manager.cc:725] Time spent opening block manager: real 0.025s user 0.004s sys 0.011s
I20260620 03:07:46.339685 11937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "235672e6f02f449f9d49b944afc8e37d"
format_stamp: "Formatted at 2026-06-20 03:07:46 on dist-test-slave-xv68"
server_key: "b3fb6c149263a22d949504948cff1b5b"
server_key_iv: "fd7ef00365fd85bba3e8379af64b376a"
server_key_version: "kuduclusterkey@0"
I20260620 03:07:46.339933 11937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:07:46.467726 11937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:07:46.481709 11937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:07:46.482072 11937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:07:46.495118 11937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:07:46.497429 11937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:07:46.497596 11937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 03:07:46.497766 11937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:07:46.497876 11937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:07:46.613430 11931 heartbeater.cc:503] Master 127.0.97.62:38503 was elected leader, sending a full tablet report...
I20260620 03:07:46.694515 11937 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39067
I20260620 03:07:46.694554 12073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39067 every 8 connection(s)
I20260620 03:07:46.696198 11937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:07:46.701215 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 11937
I20260620 03:07:46.701381 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:07:46.701934 388 external_mini_cluster.cc:1428] Setting key 99d1463eb8498807bebf2ebea6d53171
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Jun 20 03:07:46 dist-test-slave-xv68 krb5kdc[11242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:07:46.752355 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:46.700514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38503 (local address 127.0.97.2:59175)
0620 03:07:46.725219 (+ 24705us) negotiation.cc:107] Waiting for socket to connect
0620 03:07:46.725287 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 03:07:46.727677 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:07:46.729965 (+ 2288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:07:46.729978 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:07:46.730629 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:07:46.731545 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:07:46.731575 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:07:46.736732 (+ 5157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:07:46.736738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:07:46.738728 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:07:46.738743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:07:46.739985 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:07:46.739992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:07:46.740260 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:46.741029 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:07:46.741063 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:07:46.743727 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:07:46.747832 (+ 4105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:07:46.747852 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:07:46.747870 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:07:46.748389 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:07:46.748789 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:07:46.748794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:07:46.748797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:07:46.749057 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:07:46.749726 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:07:46.749734 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:07:46.750145 (+ 411us) client_negotiation.cc:769] Sending connection context
0620 03:07:46.750436 (+ 291us) client_negotiation.cc:240] Negotiation successful
0620 03:07:46.750955 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24339,"thread_start_us":27503,"threads_started":1}
I20260620 03:07:46.753471 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:46.699660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59175 (local address 127.0.97.62:38503)
0620 03:07:46.729351 (+ 29691us) server_negotiation.cc:206] Beginning negotiation
0620 03:07:46.729358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:07:46.729378 (+ 20us) server_negotiation.cc:407] Connection header received
0620 03:07:46.729439 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:07:46.729444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:07:46.729525 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:07:46.729639 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:07:46.731780 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:46.736510 (+ 4730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:46.738924 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:46.739543 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:46.739655 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:46.743985 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:07:46.743992 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:07:46.743996 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:07:46.744035 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:07:46.747652 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:46.748523 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:46.748529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:46.748533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:46.748675 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:46.749180 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:46.749187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:46.749191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:46.749519 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:07:46.749648 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 03:07:46.752856 (+ 3208us) server_negotiation.cc:299] Negotiation successful
0620 03:07:46.753118 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29551,"thread_start_us":29422,"threads_started":1}
I20260620 03:07:46.756145 12074 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38503
I20260620 03:07:46.756453 12074 heartbeater.cc:464] Registering TS with master...
I20260620 03:07:46.757361 12074 heartbeater.cc:511] Master 127.0.97.62:38503 requested a full tablet report, sending...
I20260620 03:07:46.759166 11718 ts_manager.cc:194] Registered new tserver with Master: 235672e6f02f449f9d49b944afc8e37d (127.0.97.2:39067)
I20260620 03:07:46.760386 11718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:07:46.764822 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49057/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38503
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:07:47.120072 12080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:07:47.120337 12080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:07:47.120426 12080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:07:47.128140 12080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:07:47.128235 12080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:07:47.128348 12080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:07:47.137616 12080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:49057/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:07:47.138769 12080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:07:47.140008 12080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:07:47.152992 12085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:07:47.153038 12086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:07:47.152962 12088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:07:47.153610 12080 server_base.cc:1034] running on GCE node
I20260620 03:07:47.154232 12080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:07:47.155107 12080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:07:47.156315 12080 hybrid_clock.cc:648] HybridClock initialized: now 1781924867156257 us; error 52 us; skew 500 ppm
I20260620 03:07:47.156653 12080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:07:47 dist-test-slave-xv68 krb5kdc[11242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:07:47.159603 12080 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:07:47.161005 12080 webserver.cc:460] Webserver started at http://127.0.97.3:36773/ using document root <none> and password file <none>
I20260620 03:07:47.161607 12080 fs_manager.cc:359] Metadata directory not provided
I20260620 03:07:47.161702 12080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:07:47.161942 12080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 03:07:47 dist-test-slave-xv68 krb5kdc[11242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:07:47.179630 12080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "7b40b90f74fb49f39abbccba49e50d0e"
format_stamp: "Formatted at 2026-06-20 03:07:47 on dist-test-slave-xv68"
server_key: "bb434c75e23ff148fecae0b454016bcc"
server_key_iv: "a40ae6e5ea6f831d0ec97511f2bc48d4"
server_key_version: "kuduclusterkey@0"
I20260620 03:07:47.180728 12080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7b40b90f74fb49f39abbccba49e50d0e"
format_stamp: "Formatted at 2026-06-20 03:07:47 on dist-test-slave-xv68"
server_key: "bb434c75e23ff148fecae0b454016bcc"
server_key_iv: "a40ae6e5ea6f831d0ec97511f2bc48d4"
server_key_version: "kuduclusterkey@0"
I20260620 03:07:47.188308 12080 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260620 03:07:47.205031 12095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:07:47.206745 12080 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 03:07:47.206941 12080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "7b40b90f74fb49f39abbccba49e50d0e"
format_stamp: "Formatted at 2026-06-20 03:07:47 on dist-test-slave-xv68"
server_key: "bb434c75e23ff148fecae0b454016bcc"
server_key_iv: "a40ae6e5ea6f831d0ec97511f2bc48d4"
server_key_version: "kuduclusterkey@0"
I20260620 03:07:47.207170 12080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:07:47.230281 12080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:07:47.234387 12080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:07:47.234699 12080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:07:47.235688 12080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:07:47.237035 12080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:07:47.237103 12080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:07:47.237263 12080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:07:47.237313 12080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:07:47.271987 12080 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43419
I20260620 03:07:47.272027 12207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43419 every 8 connection(s)
I20260620 03:07:47.273253 12080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:07:47.282801 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 12080
I20260620 03:07:47.282979 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:07:47.283464 388 external_mini_cluster.cc:1428] Setting key 9169665fc815db62d4e0ca9e7e2b41e6
Jun 20 03:07:47 dist-test-slave-xv68 krb5kdc[11242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:07:47.293949 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:47.276045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43543 (local address 127.0.97.62:38503)
0620 03:07:47.276489 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 03:07:47.276494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:07:47.277441 (+ 947us) server_negotiation.cc:407] Connection header received
0620 03:07:47.279211 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:07:47.279216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:07:47.279337 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:07:47.279493 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:07:47.281340 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.282088 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.283936 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.284538 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.284670 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:47.288463 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:07:47.288468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:07:47.288472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:07:47.288504 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:07:47.291220 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:47.291997 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:47.292001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:47.292003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:47.292123 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:47.292524 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:47.292528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:47.292530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:47.292802 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:07:47.292928 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 03:07:47.293544 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 03:07:47.293702 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":220,"threads_started":1}
I20260620 03:07:47.294917 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:47.276594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38503 (local address 127.0.97.3:43543)
0620 03:07:47.277277 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 03:07:47.277320 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 03:07:47.278854 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:07:47.279687 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:07:47.279695 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:07:47.280143 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:07:47.281151 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:07:47.281171 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.282258 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.282262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:07:47.283777 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:07:47.283789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.284819 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.284828 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:07:47.285075 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:47.285869 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:07:47.285903 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:07:47.288259 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:07:47.291485 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:07:47.291494 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:07:47.291508 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:07:47.291896 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:07:47.292216 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:07:47.292220 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:07:47.292222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:07:47.292377 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:07:47.292909 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:07:47.292914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:07:47.293228 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 03:07:47.293444 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 03:07:47.293798 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":277,"threads_started":1}
I20260620 03:07:47.296446 12208 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38503
I20260620 03:07:47.296749 12208 heartbeater.cc:464] Registering TS with master...
I20260620 03:07:47.297422 12208 heartbeater.cc:511] Master 127.0.97.62:38503 requested a full tablet report, sending...
I20260620 03:07:47.299113 11717 ts_manager.cc:194] Registered new tserver with Master: 7b40b90f74fb49f39abbccba49e50d0e (127.0.97.3:43419)
I20260620 03:07:47.300024 11717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43543
I20260620 03:07:47.311421 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:07:47.328428 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:47.316411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.0.97.62:38503)
0620 03:07:47.316573 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 03:07:47.316577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:07:47.317091 (+ 514us) server_negotiation.cc:407] Connection header received
0620 03:07:47.317259 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:07:47.317262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:07:47.317337 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:07:47.317419 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:07:47.318689 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.319449 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.320722 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.321104 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.321196 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:47.323581 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:07:47.323586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:07:47.323590 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:07:47.323618 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:07:47.326131 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:47.326791 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:47.326795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:47.326798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:47.326917 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:47.327237 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:47.327240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:47.327242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:47.327577 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:07:47.327752 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 03:07:47.328036 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 03:07:47.328206 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 03:07:47.332639 11717 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:07:47.340204 11717 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:07:47.369410 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:47.353501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54318 (local address 127.0.97.2:39067)
0620 03:07:47.354691 (+ 1190us) server_negotiation.cc:206] Beginning negotiation
0620 03:07:47.354704 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:07:47.355424 (+ 720us) server_negotiation.cc:407] Connection header received
0620 03:07:47.355854 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:07:47.355862 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:07:47.357434 (+ 1572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:07:47.357677 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:07:47.358948 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.362452 (+ 3504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.366637 (+ 4185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.368122 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.368264 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:47.368472 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 03:07:47.368881 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 03:07:47.369065 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":956,"thread_start_us":828,"threads_started":1}
I20260620 03:07:47.369509 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:47.353237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39067 (local address 127.0.0.1:54318)
0620 03:07:47.355262 (+ 2025us) negotiation.cc:107] Waiting for socket to connect
0620 03:07:47.355327 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 03:07:47.355586 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:07:47.358064 (+ 2478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:07:47.358074 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:07:47.358117 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:07:47.358702 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:07:47.358718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.362699 (+ 3981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.362708 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:07:47.366467 (+ 3759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:07:47.366482 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.368452 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.368459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:07:47.368630 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:47.368672 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 03:07:47.368832 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 03:07:47.369111 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1802,"mutex_wait_us":70,"thread_start_us":1649,"threads_started":1}
I20260620 03:07:47.372520 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:47.353501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44389 (local address 127.0.0.1:37748)
0620 03:07:47.364164 (+ 10663us) negotiation.cc:107] Waiting for socket to connect
0620 03:07:47.364188 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 03:07:47.364546 (+ 358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:07:47.365383 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:07:47.365388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:07:47.365412 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:07:47.365883 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:07:47.365898 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.367470 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.367478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:07:47.369825 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:07:47.369839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.371689 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.371695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:07:47.371864 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:47.371903 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 03:07:47.372009 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 03:07:47.372130 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10505,"mutex_wait_us":3,"spinlock_wait_cycles":4992,"thread_start_us":10831,"threads_started":1}
I20260620 03:07:47.372507 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:47.354049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.0.97.1:44389)
0620 03:07:47.356009 (+ 1960us) server_negotiation.cc:206] Beginning negotiation
0620 03:07:47.356018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:07:47.364310 (+ 8292us) server_negotiation.cc:407] Connection header received
0620 03:07:47.364751 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:07:47.364763 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:07:47.365041 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:07:47.365212 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:07:47.366090 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.367271 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.370013 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.371464 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.371608 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:47.371833 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 03:07:47.372055 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 03:07:47.372201 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1789,"thread_start_us":1631,"threads_started":1}
I20260620 03:07:47.373440 12009 tablet_service.cc:1467] Processing CreateTablet for tablet 4efe510e5d794592b389136c412ae1bc (DEFAULT_TABLE table=test-table [id=849df4bae21c4bee844d2f8aa88fbd71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:07:47.374130 12222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:47.356669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43419 (local address 127.0.0.1:41466)
0620 03:07:47.364410 (+ 7741us) negotiation.cc:107] Waiting for socket to connect
0620 03:07:47.364425 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:07:47.364530 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:07:47.365681 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:07:47.365686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:07:47.365714 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:07:47.366216 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:07:47.366226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.368450 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.368456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:07:47.371460 (+ 3004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:07:47.371473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.373330 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.373338 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:07:47.373517 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:47.373553 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 03:07:47.373644 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 03:07:47.373764 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7590,"mutex_wait_us":29,"thread_start_us":7166,"threads_started":1}
I20260620 03:07:47.374182 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:47.356489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41466 (local address 127.0.97.3:43419)
0620 03:07:47.365080 (+ 8591us) server_negotiation.cc:206] Beginning negotiation
0620 03:07:47.365091 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:07:47.365122 (+ 31us) server_negotiation.cc:407] Connection header received
0620 03:07:47.365222 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:07:47.365229 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:07:47.365425 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:07:47.365868 (+ 443us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:07:47.366395 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.367936 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.371616 (+ 3680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.373126 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.373253 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:47.373447 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 03:07:47.373693 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 03:07:47.373834 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8407,"thread_start_us":8252,"threads_started":1}
I20260620 03:07:47.375414 12009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4efe510e5d794592b389136c412ae1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:07:47.376307 11866 tablet_service.cc:1467] Processing CreateTablet for tablet 4efe510e5d794592b389136c412ae1bc (DEFAULT_TABLE table=test-table [id=849df4bae21c4bee844d2f8aa88fbd71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:07:47.377213 12143 tablet_service.cc:1467] Processing CreateTablet for tablet 4efe510e5d794592b389136c412ae1bc (DEFAULT_TABLE table=test-table [id=849df4bae21c4bee844d2f8aa88fbd71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:07:47.377694 11866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4efe510e5d794592b389136c412ae1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:07:47.378358 12143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4efe510e5d794592b389136c412ae1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:07:47.389214 12227 tablet_bootstrap.cc:492] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e: Bootstrap starting.
I20260620 03:07:47.391247 12228 tablet_bootstrap.cc:492] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d: Bootstrap starting.
I20260620 03:07:47.392652 12229 tablet_bootstrap.cc:492] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a: Bootstrap starting.
I20260620 03:07:47.394300 12227 tablet_bootstrap.cc:654] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e: Neither blocks nor log segments found. Creating new log.
I20260620 03:07:47.394747 12228 tablet_bootstrap.cc:654] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d: Neither blocks nor log segments found. Creating new log.
I20260620 03:07:47.395643 12227 log.cc:824] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e: Log is configured to *not* fsync() on all Append() calls
I20260620 03:07:47.396009 12228 log.cc:824] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d: Log is configured to *not* fsync() on all Append() calls
I20260620 03:07:47.396726 12229 tablet_bootstrap.cc:654] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a: Neither blocks nor log segments found. Creating new log.
I20260620 03:07:47.398082 12229 log.cc:824] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a: Log is configured to *not* fsync() on all Append() calls
I20260620 03:07:47.404551 12229 tablet_bootstrap.cc:492] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a: No bootstrap required, opened a new log
I20260620 03:07:47.404909 12229 ts_tablet_manager.cc:1397] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260620 03:07:47.405361 12227 tablet_bootstrap.cc:492] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e: No bootstrap required, opened a new log
I20260620 03:07:47.405701 12227 ts_tablet_manager.cc:1397] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20260620 03:07:47.405982 12228 tablet_bootstrap.cc:492] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d: No bootstrap required, opened a new log
I20260620 03:07:47.406289 12228 ts_tablet_manager.cc:1397] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260620 03:07:47.413484 12227 raft_consensus.cc:348] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235672e6f02f449f9d49b944afc8e37d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39067 } } peers { permanent_uuid: "d08c040bc05b4a9d92f679bb8941184a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } } peers { permanent_uuid: "7b40b90f74fb49f39abbccba49e50d0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } }
I20260620 03:07:47.413889 12227 raft_consensus.cc:374] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:07:47.414012 12227 raft_consensus.cc:729] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b40b90f74fb49f39abbccba49e50d0e, State: Initialized, Role: FOLLOWER
I20260620 03:07:47.414662 12227 consensus_queue.cc:260] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235672e6f02f449f9d49b944afc8e37d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39067 } } peers { permanent_uuid: "d08c040bc05b4a9d92f679bb8941184a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } } peers { permanent_uuid: "7b40b90f74fb49f39abbccba49e50d0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } }
I20260620 03:07:47.415755 12208 heartbeater.cc:503] Master 127.0.97.62:38503 was elected leader, sending a full tablet report...
I20260620 03:07:47.416472 12229 raft_consensus.cc:348] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235672e6f02f449f9d49b944afc8e37d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39067 } } peers { permanent_uuid: "d08c040bc05b4a9d92f679bb8941184a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } } peers { permanent_uuid: "7b40b90f74fb49f39abbccba49e50d0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } }
I20260620 03:07:47.416941 12229 raft_consensus.cc:374] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:07:47.417071 12229 raft_consensus.cc:729] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d08c040bc05b4a9d92f679bb8941184a, State: Initialized, Role: FOLLOWER
I20260620 03:07:47.417575 12227 ts_tablet_manager.cc:1428] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e: Time spent starting tablet: real 0.012s user 0.005s sys 0.005s
I20260620 03:07:47.417555 12228 raft_consensus.cc:348] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235672e6f02f449f9d49b944afc8e37d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39067 } } peers { permanent_uuid: "d08c040bc05b4a9d92f679bb8941184a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } } peers { permanent_uuid: "7b40b90f74fb49f39abbccba49e50d0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } }
I20260620 03:07:47.417981 12228 raft_consensus.cc:374] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:07:47.417793 12229 consensus_queue.cc:260] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235672e6f02f449f9d49b944afc8e37d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39067 } } peers { permanent_uuid: "d08c040bc05b4a9d92f679bb8941184a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } } peers { permanent_uuid: "7b40b90f74fb49f39abbccba49e50d0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } }
I20260620 03:07:47.418107 12228 raft_consensus.cc:729] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 235672e6f02f449f9d49b944afc8e37d, State: Initialized, Role: FOLLOWER
I20260620 03:07:47.418927 12228 consensus_queue.cc:260] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235672e6f02f449f9d49b944afc8e37d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39067 } } peers { permanent_uuid: "d08c040bc05b4a9d92f679bb8941184a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } } peers { permanent_uuid: "7b40b90f74fb49f39abbccba49e50d0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } }
I20260620 03:07:47.420598 12229 ts_tablet_manager.cc:1428] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260620 03:07:47.421542 12074 heartbeater.cc:503] Master 127.0.97.62:38503 was elected leader, sending a full tablet report...
I20260620 03:07:47.422153 12228 ts_tablet_manager.cc:1428] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260620 03:07:47.454187 12075 tablet.cc:2369] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:07:47.525017 12209 tablet.cc:2369] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:07:47.554078 11932 tablet.cc:2369] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:07:47.765684 12233 raft_consensus.cc:482] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:07:47.766010 12233 raft_consensus.cc:504] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235672e6f02f449f9d49b944afc8e37d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39067 } } peers { permanent_uuid: "d08c040bc05b4a9d92f679bb8941184a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } } peers { permanent_uuid: "7b40b90f74fb49f39abbccba49e50d0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } }
I20260620 03:07:47.767693 12233 leader_election.cc:288] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 235672e6f02f449f9d49b944afc8e37d (127.0.97.2:39067), d08c040bc05b4a9d92f679bb8941184a (127.0.97.1:44389)
I20260620 03:07:47.773196 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:47.768142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53785 (local address 127.0.97.1:44389)
0620 03:07:47.768346 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 03:07:47.768354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:07:47.768418 (+ 64us) server_negotiation.cc:407] Connection header received
0620 03:07:47.768668 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:07:47.768672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:07:47.768729 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:07:47.768823 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:07:47.769188 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.769888 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.771548 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.772309 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.772451 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:47.772553 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 03:07:47.772823 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 03:07:47.772946 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:07:47.773195 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:47.768043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44389 (local address 127.0.97.3:53785)
0620 03:07:47.768340 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 03:07:47.768358 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:07:47.768515 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:07:47.768843 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:07:47.768847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:07:47.768871 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:07:47.769052 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:07:47.769061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.770033 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.770039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:07:47.771411 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:07:47.771422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.772461 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.772467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:07:47.772662 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:47.772686 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 03:07:47.772787 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 03:07:47.772919 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":1}
I20260620 03:07:47.774243 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:47.768259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39067 (local address 127.0.97.3:45961)
0620 03:07:47.768915 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 03:07:47.768929 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 03:07:47.769086 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:07:47.769414 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:07:47.769417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:07:47.769439 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:07:47.769735 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:07:47.769743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.770718 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.770723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:07:47.772449 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:07:47.772460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.773672 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.773678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:07:47.773840 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:47.773863 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 03:07:47.773936 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 03:07:47.774024 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":388,"threads_started":1}
I20260620 03:07:47.774408 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:47.768181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45961 (local address 127.0.97.2:39067)
0620 03:07:47.768399 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 03:07:47.768406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:07:47.769018 (+ 612us) server_negotiation.cc:407] Connection header received
0620 03:07:47.769230 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:07:47.769234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:07:47.769288 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:07:47.769390 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:07:47.769885 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.770568 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.772600 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.773482 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.773610 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:47.773681 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 03:07:47.773998 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 03:07:47.774133 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 03:07:47.774255 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efe510e5d794592b389136c412ae1bc" candidate_uuid: "7b40b90f74fb49f39abbccba49e50d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d08c040bc05b4a9d92f679bb8941184a" is_pre_election: true
I20260620 03:07:47.774935 11886 raft_consensus.cc:2454] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b40b90f74fb49f39abbccba49e50d0e in term 0.
I20260620 03:07:47.775365 12029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efe510e5d794592b389136c412ae1bc" candidate_uuid: "7b40b90f74fb49f39abbccba49e50d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "235672e6f02f449f9d49b944afc8e37d" is_pre_election: true
I20260620 03:07:47.775627 12099 leader_election.cc:302] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b40b90f74fb49f39abbccba49e50d0e, d08c040bc05b4a9d92f679bb8941184a; no voters:
I20260620 03:07:47.775913 12029 raft_consensus.cc:2454] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b40b90f74fb49f39abbccba49e50d0e in term 0.
I20260620 03:07:47.776078 12233 raft_consensus.cc:2781] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:07:47.776217 12233 raft_consensus.cc:482] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:07:47.776325 12233 raft_consensus.cc:3037] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:07:47.778520 12233 raft_consensus.cc:504] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235672e6f02f449f9d49b944afc8e37d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39067 } } peers { permanent_uuid: "d08c040bc05b4a9d92f679bb8941184a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } } peers { permanent_uuid: "7b40b90f74fb49f39abbccba49e50d0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } }
I20260620 03:07:47.779112 12233 leader_election.cc:288] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [CANDIDATE]: Term 1 election: Requested vote from peers 235672e6f02f449f9d49b944afc8e37d (127.0.97.2:39067), d08c040bc05b4a9d92f679bb8941184a (127.0.97.1:44389)
I20260620 03:07:47.779503 12029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efe510e5d794592b389136c412ae1bc" candidate_uuid: "7b40b90f74fb49f39abbccba49e50d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "235672e6f02f449f9d49b944afc8e37d"
I20260620 03:07:47.779733 12029 raft_consensus.cc:3037] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:07:47.779729 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efe510e5d794592b389136c412ae1bc" candidate_uuid: "7b40b90f74fb49f39abbccba49e50d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d08c040bc05b4a9d92f679bb8941184a"
I20260620 03:07:47.779958 11886 raft_consensus.cc:3037] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:07:47.783001 12029 raft_consensus.cc:2454] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b40b90f74fb49f39abbccba49e50d0e in term 1.
I20260620 03:07:47.783073 11886 raft_consensus.cc:2454] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b40b90f74fb49f39abbccba49e50d0e in term 1.
I20260620 03:07:47.783439 12099 leader_election.cc:302] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b40b90f74fb49f39abbccba49e50d0e, d08c040bc05b4a9d92f679bb8941184a; no voters:
I20260620 03:07:47.783740 12233 raft_consensus.cc:2781] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:07:47.784186 12233 raft_consensus.cc:686] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [term 1 LEADER]: Becoming Leader. State: Replica: 7b40b90f74fb49f39abbccba49e50d0e, State: Running, Role: LEADER
I20260620 03:07:47.784693 12233 consensus_queue.cc:237] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235672e6f02f449f9d49b944afc8e37d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39067 } } peers { permanent_uuid: "d08c040bc05b4a9d92f679bb8941184a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } } peers { permanent_uuid: "7b40b90f74fb49f39abbccba49e50d0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } }
I20260620 03:07:47.789451 11717 catalog_manager.cc:5495] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b40b90f74fb49f39abbccba49e50d0e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7b40b90f74fb49f39abbccba49e50d0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235672e6f02f449f9d49b944afc8e37d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d08c040bc05b4a9d92f679bb8941184a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b40b90f74fb49f39abbccba49e50d0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } health_report { overall_health: HEALTHY } } }
I20260620 03:07:47.820088 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:47.814624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47602 (local address 127.0.97.3:43419)
0620 03:07:47.814828 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 03:07:47.814834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:07:47.814911 (+ 77us) server_negotiation.cc:407] Connection header received
0620 03:07:47.815165 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:07:47.815170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:07:47.815244 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:07:47.815446 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:07:47.815801 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.816522 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.817938 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.818321 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.818474 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:47.818988 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:07:47.819543 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:07:47.819700 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 03:07:47.819749 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 03:07:47.819822 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:07:47.827834 11886 raft_consensus.cc:1261] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a [term 1 FOLLOWER]: Refusing update from remote peer 7b40b90f74fb49f39abbccba49e50d0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:07:47.827848 12029 raft_consensus.cc:1261] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d [term 1 FOLLOWER]: Refusing update from remote peer 7b40b90f74fb49f39abbccba49e50d0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:07:47.828794 12237 consensus_queue.cc:1035] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "235672e6f02f449f9d49b944afc8e37d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:07:47.829213 12233 consensus_queue.cc:1035] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d08c040bc05b4a9d92f679bb8941184a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:07:47.855211 11717 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:07:47.855553 11717 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:07:47.858731 11717 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71ee6b246d78456dbfa7242f6360dd2f: Sending DeleteTablet for 3 replicas of tablet 4efe510e5d794592b389136c412ae1bc
I20260620 03:07:47.859993 12009 tablet_service.cc:1514] Processing DeleteTablet for tablet 4efe510e5d794592b389136c412ae1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:07:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54318
I20260620 03:07:47.860512 12143 tablet_service.cc:1514] Processing DeleteTablet for tablet 4efe510e5d794592b389136c412ae1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:07:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41466
I20260620 03:07:47.861171 11866 tablet_service.cc:1514] Processing DeleteTablet for tablet 4efe510e5d794592b389136c412ae1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:07:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37748
I20260620 03:07:47.862167 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 11798
I20260620 03:07:47.865581 12252 tablet_replica.cc:330] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a: stopping tablet replica
I20260620 03:07:47.866155 12252 raft_consensus.cc:2229] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:07:47.866850 12251 tablet_replica.cc:330] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e: stopping tablet replica
I20260620 03:07:47.867024 12250 tablet_replica.cc:330] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d: stopping tablet replica
I20260620 03:07:47.867468 12250 raft_consensus.cc:2229] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:07:47.867620 12251 raft_consensus.cc:2229] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:07:47.867844 12252 raft_consensus.cc:2258] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:07:47.868010 12250 raft_consensus.cc:2258] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:07:47.868753 12251 raft_consensus.cc:2258] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:07:47.870096 12250 ts_tablet_manager.cc:1905] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:07:47.870429 12252 ts_tablet_manager.cc:1905] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:07:47.874910 12251 ts_tablet_manager.cc:1905] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:07:47.877106 12250 ts_tablet_manager.cc:1918] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:07:47.877321 12250 log.cc:1196] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/4efe510e5d794592b389136c412ae1bc
I20260620 03:07:47.877861 12250 ts_tablet_manager.cc:1939] T 4efe510e5d794592b389136c412ae1bc P 235672e6f02f449f9d49b944afc8e37d: Deleting consensus metadata
I20260620 03:07:47.879668 11703 catalog_manager.cc:4841] TS 235672e6f02f449f9d49b944afc8e37d (127.0.97.2:39067): tablet 4efe510e5d794592b389136c412ae1bc (table test-table [id=849df4bae21c4bee844d2f8aa88fbd71]) successfully deleted
Jun 20 03:07:47 dist-test-slave-xv68 krb5kdc[11242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:07:47.883136 12251 ts_tablet_manager.cc:1918] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:07:47.883361 12251 log.cc:1196] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/4efe510e5d794592b389136c412ae1bc
I20260620 03:07:47.883800 12252 ts_tablet_manager.cc:1918] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:07:47.883917 12251 ts_tablet_manager.cc:1939] T 4efe510e5d794592b389136c412ae1bc P 7b40b90f74fb49f39abbccba49e50d0e: Deleting consensus metadata
I20260620 03:07:47.884001 12252 log.cc:1196] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/4efe510e5d794592b389136c412ae1bc
I20260620 03:07:47.884670 12252 ts_tablet_manager.cc:1939] T 4efe510e5d794592b389136c412ae1bc P d08c040bc05b4a9d92f679bb8941184a: Deleting consensus metadata
I20260620 03:07:47.885560 11703 catalog_manager.cc:4841] TS 7b40b90f74fb49f39abbccba49e50d0e (127.0.97.3:43419): tablet 4efe510e5d794592b389136c412ae1bc (table test-table [id=849df4bae21c4bee844d2f8aa88fbd71]) successfully deleted
I20260620 03:07:47.886502 11705 catalog_manager.cc:4841] TS d08c040bc05b4a9d92f679bb8941184a (127.0.97.1:44389): tablet 4efe510e5d794592b389136c412ae1bc (table test-table [id=849df4bae21c4bee844d2f8aa88fbd71]) successfully deleted
I20260620 03:07:47.889643 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:47.863474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.0.97.1:44389)
0620 03:07:47.863896 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 03:07:47.863905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:07:47.870486 (+ 6581us) server_negotiation.cc:407] Connection header received
0620 03:07:47.870737 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:07:47.870742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:07:47.870819 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:07:47.870893 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:07:47.872715 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.873838 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.875868 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:47.876524 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:47.876627 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:47.882259 (+ 5632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:07:47.882273 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:07:47.882289 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:07:47.882329 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:07:47.886850 (+ 4521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:47.887607 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:47.887622 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:47.887625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:47.887747 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:47.888142 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:47.888147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:47.888149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:47.888469 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:07:47.888793 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 03:07:47.889108 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 03:07:47.889306 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260620 03:07:47.890308 11926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:07:48.016553 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 11798
I20260620 03:07:48.033810 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 11937
Jun 20 03:07:48 dist-test-slave-xv68 krb5kdc[11242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:07:48.056375 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:48.034626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.0.97.2:39067)
0620 03:07:48.034773 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 03:07:48.034777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:07:48.034795 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:07:48.034951 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:07:48.034955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:07:48.035035 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:07:48.035116 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:07:48.036747 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:48.037500 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:48.039546 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:48.040163 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:48.040284 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:48.045791 (+ 5507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:07:48.045810 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:07:48.045831 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:07:48.045880 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:07:48.049716 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:48.050611 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:48.050621 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:48.050625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:48.050794 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:48.053706 (+ 2912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:48.053712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:48.053715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:48.054224 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:07:48.054540 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 03:07:48.055762 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0620 03:07:48.055992 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 03:07:48.057106 12069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:07:48.148079 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 11937
I20260620 03:07:48.165345 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 12080
Jun 20 03:07:48 dist-test-slave-xv68 krb5kdc[11242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:07:48.184226 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:07:48.166187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47610 (local address 127.0.97.3:43419)
0620 03:07:48.166378 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 03:07:48.166390 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:07:48.166655 (+ 265us) server_negotiation.cc:407] Connection header received
0620 03:07:48.166857 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:07:48.166862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:07:48.166935 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:07:48.167020 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:07:48.168767 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:48.169552 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:48.171587 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:07:48.172275 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:07:48.172392 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:07:48.177985 (+ 5593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:07:48.177998 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:07:48.178013 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:07:48.178048 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:07:48.181306 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:48.182198 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:48.182205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:48.182207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:48.182348 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:07:48.182796 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:07:48.182799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:07:48.182801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:07:48.183122 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:07:48.183427 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 03:07:48.183762 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 03:07:48.183938 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 03:07:48.185153 12203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:07:48.307356 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 12080
I20260620 03:07:48.326583 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 11686
I20260620 03:07:48.327845 11748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:07:48.440328 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 11686
I20260620 03:07:48.464913 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 03:07:48.800061 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 03:07:48.800282 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 03:07:49.145737 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 03:07:49.146 UTC [11272] LOG: received smart shutdown request
2026-06-20 03:07:49.149 UTC [11272] LOG: background worker "logical replication launcher" (PID 11281) exited with exit code 1
2026-06-20 03:07:49.149 UTC [11276] LOG: shutting down
2026-06-20 03:07:49.319 UTC [11272] LOG: database system is shut down
2026-06-20T03:07:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48824 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:07:49 dist-test-slave-xv68 krb5kdc[12266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:07:49 dist-test-slave-xv68 krb5kdc[12266](info): set up 2 sockets
Jun 20 03:07:49 dist-test-slave-xv68 krb5kdc[12266](info): commencing operation
krb5kdc: starting...
W20260620 03:07:50.559682 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:07:50 dist-test-slave-xv68 krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:07:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:07:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 03:07:50.867655 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5VLrcV/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/postgres -l logfile start
2026-06-20 03:07:53.420 UTC [12296] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 03:07:53.420 UTC [12296] LOG: listening on IPv4 address "127.0.97.20", port 38841
2026-06-20 03:07:53.423 UTC [12296] LOG: listening on Unix socket "/tmp/.s.PGSQL.38841"
2026-06-20 03:07:53.445 UTC [12299] LOG: database system was shut down at 2026-06-20 03:07:51 UTC
2026-06-20 03:07:53.450 UTC [12296] LOG: database system is ready to accept connections
I20260620 03:07:54.443333 388 mini_postgres.cc:96] Postgres bound to 38841
2026-06-20 03:07:54.460 UTC [12307] FATAL: database "slave" does not exist
127.0.97.20:38841 - accepting connections
I20260620 03:07:54.461078 388 mini_ranger.cc:161] Starting Ranger...
I20260620 03:07:54.491750 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 03:07:54.998761 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 03:07:54.998896 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-admin
2026-06-20 03:07:55,282 [I] DB FLAVOR :POSTGRES
2026-06-20 03:07:55,282 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 03:07:55,282 [I] Checking connection..
2026-06-20 03:07:55,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 03:07:55,532 [I] Checking connection passed.
2026-06-20 03:07:55,533 [I] --------- Verifying version history table ---------
2026-06-20 03:07:55,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 03:07:55,811 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 03:07:55,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 03:07:56,072 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 03:07:56,072 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 03:07:56,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 03:07:56.298 UTC [12371] WARNING: there is no transaction in progress
2026-06-20 03:07:56,305 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 03:07:56,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 03:07:56,533 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 03:07:56,533 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 03:07:56,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 03:07:56,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 03:07:57,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-xv68.c.gcp-upstream.internal','N') ;"
2026-06-20 03:07:57,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 03:07:57,496 [I] Table x_portal_user does not exist in database ranger
2026-06-20 03:07:57,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 03:07:57,753 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 03:07:57,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 03:07:57,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 03:07:58,249 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 03:07:58,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 03:07:59.222 UTC [12479] WARNING: there is no transaction in progress
2026-06-20 03:07:59.239 UTC [12479] WARNING: there is no transaction in progress
2026-06-20 03:07:59.256 UTC [12479] WARNING: there is no transaction in progress
2026-06-20 03:07:59.272 UTC [12479] WARNING: there is no transaction in progress
2026-06-20 03:07:59.290 UTC [12479] WARNING: there is no transaction in progress
2026-06-20 03:07:59.310 UTC [12479] WARNING: there is no transaction in progress
2026-06-20 03:07:59.365 UTC [12479] WARNING: there is no transaction in progress
2026-06-20 03:07:59.372 UTC [12479] WARNING: there is no transaction in progress
2026-06-20 03:07:59.387 UTC [12479] WARNING: there is no transaction in progress
2026-06-20 03:07:59.400 UTC [12479] WARNING: there is no transaction in progress
2026-06-20 03:07:59.411 UTC [12479] WARNING: there is no transaction in progress
2026-06-20 03:07:59.425 UTC [12479] WARNING: there is no transaction in progress
2026-06-20 03:07:59.435 UTC [12479] WARNING: there is no transaction in progress
2026-06-20 03:07:59.441 UTC [12479] WARNING: there is no transaction in progress
2026-06-20 03:07:59.447 UTC [12479] WARNING: there is no transaction in progress
2026-06-20 03:07:59.453 UTC [12479] WARNING: there is no transaction in progress
2026-06-20 03:08:00,107 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 03:08:00,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 03:08:00,342 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 03:08:00,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 03:08:00,584 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 03:08:00,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 03:08:00,847 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 03:08:00,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 03:08:01,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 03:08:01,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-xv68.c.gcp-upstream.internal';"
2026-06-20 03:08:01,591 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 03:08:01,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 03:08:01,828 [I] DB_PATCHES have already been applied
I20260620 03:08:01.834419 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5VLrcV/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 03:08:01.834502 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 03:08:01.848690 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:36973
Jun 20, 2026 3:08:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:08:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:08:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 3:08:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 3:08:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 3:08:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 3:08:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 03:08:02.982813 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.522s user 0.008s sys 0.033s
Jun 20, 2026 3:08:22 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 3:08:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 3:08:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 3:08:22 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 3:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 3:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 3:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 3:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 3:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 3:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 3:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 3:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 3:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 3:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 3:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 3:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 3:08:22 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 3:08:23 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 03:08:23.705559 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 03:08:23 dist-test-slave-xv68 krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924903, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 03:08:23.862934 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 03:08:23.884888 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 03:08:24.733551 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 03:08:24.733680 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-kms
I20260620 03:08:24.733728 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:38841
2026-06-20 03:08:25,088 [I] DB FLAVOR :POSTGRES
2026-06-20 03:08:25,088 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 03:08:25,088 [I] Checking connection
2026-06-20 03:08:25,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 03:08:25,354 [I] connection success
2026-06-20 03:08:25,354 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 03:08:25,355 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 03:08:25,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 03:08:25,598 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 03:08:25,598 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 03:08:25,598 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 03:08:25,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 03:08:25,855 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 03:08:26.363128 388 mini_ranger_kms.cc:310] Created kms service
I20260620 03:08:26.594465 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 03:08:26.635820 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 03:08:26 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924903, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:08:27.004357 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 03:08:27.004725 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task5VLrcV/build/asan/bin/postgresql.jar:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5VLrcV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 03:08:27.004802 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 03:08:27.019501 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12659
I20260620 03:08:27.019632 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:45741
Jun 20, 2026 3:08:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:08:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:08:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 3:08:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 3:08:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 3:08:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 03:08:29 dist-test-slave-xv68 krb5kdc[12266](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781924909, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:29.303707 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.441s user 0.075s sys 0.291s
I20260620 03:08:29.303956 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 03:08:29.304121 388 mini_ranger_kms.cc:401] 127.0.97.20:45741/kms/v1/keys
Jun 20 03:08:29 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924909, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 03:08:29 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924909, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 03:08:32 dist-test-slave-xv68 krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:08:32.152509 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41153
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41153
--ranger_config_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45741/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:08:32.483254 12710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:32.483621 12710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:32.483712 12710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:32.491492 12710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:08:32.491617 12710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:32.491662 12710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:32.491712 12710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:08:32.491761 12710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:08:32.502166 12710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:45741/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41153
--ranger_config_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:32.503603 12710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:32.504932 12710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:32.516852 12716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:32.516827 12718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:32.517196 12715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:32.518070 12710 server_base.cc:1034] running on GCE node
I20260620 03:08:32.518893 12710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:32.520357 12710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:32.521588 12710 hybrid_clock.cc:648] HybridClock initialized: now 1781924912521547 us; error 54 us; skew 500 ppm
I20260620 03:08:32.521930 12710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:32 dist-test-slave-xv68 krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:32.524920 12710 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:08:32.526294 12710 webserver.cc:460] Webserver started at http://127.0.97.62:45973/ using document root <none> and password file <none>
I20260620 03:08:32.526857 12710 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:32.526950 12710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:32.527179 12710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 03:08:32 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:08:32.587502 12710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "45885db77bf8404fb8a6dc5535f6f794"
format_stamp: "Formatted at 2026-06-20 03:08:32 on dist-test-slave-xv68"
server_key: "36bdca808b2ddfbcc2226c57c5805291"
server_key_iv: "dfc8a7a08e278d4605b0cb3905b2e003"
server_key_version: "kuduclusterkey@0"
I20260620 03:08:32.588444 12710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "45885db77bf8404fb8a6dc5535f6f794"
format_stamp: "Formatted at 2026-06-20 03:08:32 on dist-test-slave-xv68"
server_key: "36bdca808b2ddfbcc2226c57c5805291"
server_key_iv: "dfc8a7a08e278d4605b0cb3905b2e003"
server_key_version: "kuduclusterkey@0"
I20260620 03:08:32.594245 12710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:08:32.610533 12725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:32.612167 12710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 03:08:32.612352 12710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "45885db77bf8404fb8a6dc5535f6f794"
format_stamp: "Formatted at 2026-06-20 03:08:32 on dist-test-slave-xv68"
server_key: "36bdca808b2ddfbcc2226c57c5805291"
server_key_iv: "dfc8a7a08e278d4605b0cb3905b2e003"
server_key_version: "kuduclusterkey@0"
I20260620 03:08:32.612562 12710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:32.635100 12710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:32.639475 12710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:32.639849 12710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:32.660419 12710 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41153
I20260620 03:08:32.660442 12776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41153 every 8 connection(s)
I20260620 03:08:32.661819 12710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:08:32.666102 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 12710
I20260620 03:08:32.666162 12777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:08:32.666350 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:08:32.667326 388 external_mini_cluster.cc:1428] Setting key 1c97e0aaa107f596e808467defaa78bb
I20260620 03:08:32.684330 12777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794: Bootstrap starting.
I20260620 03:08:32.688045 12777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794: Neither blocks nor log segments found. Creating new log.
I20260620 03:08:32.689342 12777 log.cc:824] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:32.692909 12777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794: No bootstrap required, opened a new log
Jun 20 03:08:32 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:08:32.703735 12777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45885db77bf8404fb8a6dc5535f6f794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41153 } }
I20260620 03:08:32.704209 12777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:08:32.704313 12777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45885db77bf8404fb8a6dc5535f6f794, State: Initialized, Role: FOLLOWER
I20260620 03:08:32.704998 12777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45885db77bf8404fb8a6dc5535f6f794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41153 } }
I20260620 03:08:32.705266 12777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:08:32.705389 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:32.669751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.0.97.62:41153)
0620 03:08:32.680332 (+ 10581us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:32.680352 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:32.680404 (+ 52us) server_negotiation.cc:407] Connection header received
0620 03:08:32.683038 (+ 2634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:32.683069 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:32.683654 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:32.684109 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:32.685763 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:32.687839 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:32.689720 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:32.690788 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:32.690968 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:32.696027 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:32.696042 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:32.696063 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:32.696109 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:32.700302 (+ 4193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:32.701166 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:32.701176 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:32.701195 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:32.701405 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:32.701855 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:32.701861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:32.701864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:32.702615 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:32.702891 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:32.703629 (+ 738us) server_negotiation.cc:299] Negotiation successful
0620 03:08:32.704048 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10208,"thread_start_us":9949,"threads_started":1}
I20260620 03:08:32.705475 12777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:08:32.705691 12777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:08:32.708734 12777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45885db77bf8404fb8a6dc5535f6f794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41153 } }
I20260620 03:08:32.709252 12777 leader_election.cc:302] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45885db77bf8404fb8a6dc5535f6f794; no voters:
I20260620 03:08:32.709852 12777 leader_election.cc:288] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:08:32.709993 12782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:08:32.710618 12782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [term 1 LEADER]: Becoming Leader. State: Replica: 45885db77bf8404fb8a6dc5535f6f794, State: Running, Role: LEADER
I20260620 03:08:32.711262 12782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45885db77bf8404fb8a6dc5535f6f794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41153 } }
I20260620 03:08:32.713174 12782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45885db77bf8404fb8a6dc5535f6f794. Latest consensus state: current_term: 1 leader_uuid: "45885db77bf8404fb8a6dc5535f6f794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45885db77bf8404fb8a6dc5535f6f794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41153 } } }
I20260620 03:08:32.715322 12782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [sys.catalog]: This master's current role is: LEADER
I20260620 03:08:32.716051 12784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45885db77bf8404fb8a6dc5535f6f794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45885db77bf8404fb8a6dc5535f6f794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41153 } } }
I20260620 03:08:32.716529 12784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [sys.catalog]: This master's current role is: LEADER
I20260620 03:08:32.718760 12777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:08:32.727870 12786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:08:32.734314 12786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:08:32.754011 12786 catalog_manager.cc:1347] Generated new cluster ID: 794e62162aab4a25b041b2a60863a144
I20260620 03:08:32.754136 12786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:08:32.776443 12777 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 03:08:32.783943 12786 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:08:32.785869 12786 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:08:32.821149 12786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794: Generated new TSK 0
I20260620 03:08:32.822330 12786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:08:33.505734 12819 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:08:33.611199 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45741/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41153
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:08:33 dist-test-slave-xv68 krb5kdc[12266](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781924913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 03:08:34.048801 12822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:34.049291 12822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:34.049492 12822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:34.060563 12822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:34.060714 12822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:34.060909 12822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:08:34.076066 12822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:45741/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:34.077486 12822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:34.078847 12822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:34.113062 12833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:34.113086 12832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:34.123044 12822 server_base.cc:1034] running on GCE node
W20260620 03:08:34.124049 12835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:34.124917 12822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:34.127260 12822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:34.128631 12822 hybrid_clock.cc:648] HybridClock initialized: now 1781924914128541 us; error 75 us; skew 500 ppm
I20260620 03:08:34.129058 12822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:34 dist-test-slave-xv68 krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:34.133589 12822 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:08:34.145890 12822 webserver.cc:460] Webserver started at http://127.0.97.1:43007/ using document root <none> and password file <none>
I20260620 03:08:34.146859 12822 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:34.147006 12822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:34.147382 12822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 03:08:34 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:08:34.167526 12822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "e360b58de2934d0e90cf908159b0b5ff"
format_stamp: "Formatted at 2026-06-20 03:08:34 on dist-test-slave-xv68"
server_key: "8b4f2cfe58fec582a7fc4ca3268302d6"
server_key_iv: "0d94172b9962218830f12bcec93a9ef0"
server_key_version: "kuduclusterkey@0"
I20260620 03:08:34.168583 12822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e360b58de2934d0e90cf908159b0b5ff"
format_stamp: "Formatted at 2026-06-20 03:08:34 on dist-test-slave-xv68"
server_key: "8b4f2cfe58fec582a7fc4ca3268302d6"
server_key_iv: "0d94172b9962218830f12bcec93a9ef0"
server_key_version: "kuduclusterkey@0"
I20260620 03:08:34.181934 12822 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.005s sys 0.007s
I20260620 03:08:34.219697 12842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:34.221884 12822 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.007s sys 0.006s
I20260620 03:08:34.222100 12822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "e360b58de2934d0e90cf908159b0b5ff"
format_stamp: "Formatted at 2026-06-20 03:08:34 on dist-test-slave-xv68"
server_key: "8b4f2cfe58fec582a7fc4ca3268302d6"
server_key_iv: "0d94172b9962218830f12bcec93a9ef0"
server_key_version: "kuduclusterkey@0"
I20260620 03:08:34.222385 12822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:34.246053 12822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:34.251231 12822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:34.251596 12822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:34.263737 12822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
Jun 20 03:08:34 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:08:34.266376 12822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:08:34.266474 12822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:34.266594 12822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:08:34.266655 12822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:34.390865 12822 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37675
I20260620 03:08:34.390892 12954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37675 every 8 connection(s)
I20260620 03:08:34.392515 12822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:08:34.402382 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 12822
I20260620 03:08:34.402644 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:08:34.403566 388 external_mini_cluster.cc:1428] Setting key a16506d472d4efa88dd666890ca928fc
Jun 20 03:08:34 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:08:34.423874 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:34.396046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41153 (local address 127.0.97.1:58825)
0620 03:08:34.396903 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:34.396949 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:34.400147 (+ 3198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:34.404441 (+ 4294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:34.404454 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:34.405066 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:34.405849 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:34.405872 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:34.407403 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:34.407409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:34.409472 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:34.409487 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:34.410692 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:34.410699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:34.410993 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:34.411775 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:34.411805 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:34.414513 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:34.418923 (+ 4410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:34.418933 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:34.418951 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:34.419535 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:34.419971 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:34.419979 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:34.419986 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:34.420262 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:34.420960 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:34.420971 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:34.421487 (+ 516us) client_negotiation.cc:769] Sending connection context
0620 03:08:34.421772 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 03:08:34.422368 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":578,"threads_started":1}
I20260620 03:08:34.428475 12958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:34.396554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58825 (local address 127.0.97.62:41153)
0620 03:08:34.403683 (+ 7129us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:34.403690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:34.403710 (+ 20us) server_negotiation.cc:407] Connection header received
0620 03:08:34.403772 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:34.403777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:34.403886 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:34.404002 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:34.406064 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:34.407191 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:34.409653 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:34.410296 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:34.410409 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:34.414770 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:34.414776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:34.414782 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:34.414835 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:34.418752 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:34.419670 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:34.419679 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:34.419686 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:34.419836 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:34.420432 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:34.420439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:34.420446 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:34.420819 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:34.424363 (+ 3544us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:34.427762 (+ 3399us) server_negotiation.cc:299] Negotiation successful
0620 03:08:34.428026 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6988,"thread_start_us":7326,"threads_started":1}
I20260620 03:08:34.430230 12955 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41153
I20260620 03:08:34.430557 12955 heartbeater.cc:464] Registering TS with master...
I20260620 03:08:34.431438 12955 heartbeater.cc:511] Master 127.0.97.62:41153 requested a full tablet report, sending...
I20260620 03:08:34.433913 12742 ts_manager.cc:194] Registered new tserver with Master: e360b58de2934d0e90cf908159b0b5ff (127.0.97.1:37675)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 03:08:34.435994 12742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:08:34.474273 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45741/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41153
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:08:34 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 03:08:34.944747 12961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:34.945096 12961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:34.945269 12961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:34.952840 12961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:34.953014 12961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:34.953202 12961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:08:34.964288 12961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:45741/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:34.965883 12961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:34.967399 12961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:34.984843 12975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:34.985183 12976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:34.984928 12978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:34.985783 12961 server_base.cc:1034] running on GCE node
I20260620 03:08:34.986516 12961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:34.987547 12961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:34.988785 12961 hybrid_clock.cc:648] HybridClock initialized: now 1781924914988717 us; error 49 us; skew 500 ppm
I20260620 03:08:34.989105 12961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:34 dist-test-slave-xv68 krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:34.992463 12961 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:08:34.994155 12961 webserver.cc:460] Webserver started at http://127.0.97.2:40311/ using document root <none> and password file <none>
I20260620 03:08:34.994870 12961 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:34.995019 12961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:34.995424 12961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 03:08:34 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:08:35.012718 12961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "2c831e7bb0c440299b2482eb8695facf"
format_stamp: "Formatted at 2026-06-20 03:08:35 on dist-test-slave-xv68"
server_key: "3fa64d9fe4395f5cf8c5825cc763dd50"
server_key_iv: "4009f9e80fd880c17458cb156caef8be"
server_key_version: "kuduclusterkey@0"
I20260620 03:08:35.013578 12961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2c831e7bb0c440299b2482eb8695facf"
format_stamp: "Formatted at 2026-06-20 03:08:35 on dist-test-slave-xv68"
server_key: "3fa64d9fe4395f5cf8c5825cc763dd50"
server_key_iv: "4009f9e80fd880c17458cb156caef8be"
server_key_version: "kuduclusterkey@0"
I20260620 03:08:35.019415 12961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 03:08:35.043951 12985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:35.046037 12961 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.010s sys 0.001s
I20260620 03:08:35.046239 12961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "2c831e7bb0c440299b2482eb8695facf"
format_stamp: "Formatted at 2026-06-20 03:08:35 on dist-test-slave-xv68"
server_key: "3fa64d9fe4395f5cf8c5825cc763dd50"
server_key_iv: "4009f9e80fd880c17458cb156caef8be"
server_key_version: "kuduclusterkey@0"
I20260620 03:08:35.046515 12961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:35.067798 12961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:35.073670 12961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:35.074029 12961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:35.075328 12961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:08:35.077117 12961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:08:35.077217 12961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:35.077394 12961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:08:35.077474 12961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:35.116057 12961 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35187
I20260620 03:08:35.116137 13097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35187 every 8 connection(s)
I20260620 03:08:35.117770 12961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:08:35.126446 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 12961
I20260620 03:08:35.126633 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:08:35.127185 388 external_mini_cluster.cc:1428] Setting key 158c67b5ce137576d2efa876ed49f77a
Jun 20 03:08:35 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:08:35.161264 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:35.121352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41153 (local address 127.0.97.2:48647)
0620 03:08:35.125397 (+ 4045us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:35.125476 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:35.128208 (+ 2732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:35.139604 (+ 11396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:35.139619 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:35.140345 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:35.141270 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:35.141299 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:35.143427 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:35.143432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:35.144866 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:35.144886 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:35.145720 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:35.145727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:35.145977 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:35.146728 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:35.146758 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:35.149851 (+ 3093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:35.155426 (+ 5575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:35.155437 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:35.155454 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:35.156080 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:35.156567 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:35.156572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:35.156575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:35.156856 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:35.158171 (+ 1315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:35.158179 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:35.158665 (+ 486us) client_negotiation.cc:769] Sending connection context
0620 03:08:35.158989 (+ 324us) client_negotiation.cc:240] Negotiation successful
0620 03:08:35.159723 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3652,"thread_start_us":3401,"threads_started":1}
I20260620 03:08:35.162588 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:35.120780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48647 (local address 127.0.97.62:41153)
0620 03:08:35.138753 (+ 17973us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:35.138762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:35.138794 (+ 32us) server_negotiation.cc:407] Connection header received
0620 03:08:35.138883 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:35.138892 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:35.139028 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:35.139204 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:35.141500 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:35.143065 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:35.145048 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:35.145480 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:35.145626 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:35.150134 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:35.150141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:35.150147 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:35.150190 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:35.153752 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:35.156239 (+ 2487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:35.156245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:35.156248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:35.156417 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:35.156997 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:35.157002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:35.157005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:35.157355 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:35.158086 (+ 731us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:35.161905 (+ 3819us) server_negotiation.cc:299] Negotiation successful
0620 03:08:35.162207 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17788,"thread_start_us":17593,"threads_started":1}
I20260620 03:08:35.164568 13098 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41153
I20260620 03:08:35.164916 13098 heartbeater.cc:464] Registering TS with master...
I20260620 03:08:35.165849 13098 heartbeater.cc:511] Master 127.0.97.62:41153 requested a full tablet report, sending...
I20260620 03:08:35.167723 12742 ts_manager.cc:194] Registered new tserver with Master: 2c831e7bb0c440299b2482eb8695facf (127.0.97.2:35187)
I20260620 03:08:35.168833 12742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48647
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:08:35.213073 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45741/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41153
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:08:35.440085 12955 heartbeater.cc:503] Master 127.0.97.62:41153 was elected leader, sending a full tablet report...
W20260620 03:08:35.579640 13104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:35.579936 13104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:35.580122 13104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:35.587689 13104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:35.587808 13104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:35.588030 13104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:08:35.598227 13104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:45741/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:35.599668 13104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:35.600975 13104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:35.614370 13110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:35.614389 13112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:35.614457 13109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:35.616879 13104 server_base.cc:1034] running on GCE node
I20260620 03:08:35.617439 13104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:35.618374 13104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:35.619603 13104 hybrid_clock.cc:648] HybridClock initialized: now 1781924915619542 us; error 56 us; skew 500 ppm
I20260620 03:08:35.619967 13104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:35 dist-test-slave-xv68 krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:35.623027 13104 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:08:35.624564 13104 webserver.cc:460] Webserver started at http://127.0.97.3:44801/ using document root <none> and password file <none>
I20260620 03:08:35.625247 13104 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:35.625389 13104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:35.625697 13104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 03:08:35 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:08:35.639748 13104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "303a3ced4f8541ae9bcc3b25611bec62"
format_stamp: "Formatted at 2026-06-20 03:08:35 on dist-test-slave-xv68"
server_key: "56a1d5ef53355430307cc7b45ad7a299"
server_key_iv: "0b4a6670c9b79f0532c0ea53597e68b9"
server_key_version: "kuduclusterkey@0"
I20260620 03:08:35.640558 13104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "303a3ced4f8541ae9bcc3b25611bec62"
format_stamp: "Formatted at 2026-06-20 03:08:35 on dist-test-slave-xv68"
server_key: "56a1d5ef53355430307cc7b45ad7a299"
server_key_iv: "0b4a6670c9b79f0532c0ea53597e68b9"
server_key_version: "kuduclusterkey@0"
I20260620 03:08:35.645782 13104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:08:35.659127 13119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:35.660813 13104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:08:35.661012 13104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "303a3ced4f8541ae9bcc3b25611bec62"
format_stamp: "Formatted at 2026-06-20 03:08:35 on dist-test-slave-xv68"
server_key: "56a1d5ef53355430307cc7b45ad7a299"
server_key_iv: "0b4a6670c9b79f0532c0ea53597e68b9"
server_key_version: "kuduclusterkey@0"
I20260620 03:08:35.661244 13104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:35.686605 13104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:35.690706 13104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:35.691028 13104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:35.692077 13104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:08:35.693534 13104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:08:35.693638 13104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:35.693786 13104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:08:35.693858 13104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:35.729032 13104 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45325
I20260620 03:08:35.729113 13231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45325 every 8 connection(s)
I20260620 03:08:35.730345 13104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:08:35.732590 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 13104
I20260620 03:08:35.732784 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:08:35.733345 388 external_mini_cluster.cc:1428] Setting key 7c8bffc5791f7e1a1a56ed9e70fd88b3
Jun 20 03:08:35 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:08:35.750389 13234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:35.732899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41861 (local address 127.0.97.62:41153)
0620 03:08:35.733550 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:35.733555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:35.734623 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 03:08:35.736921 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:35.736926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:35.736989 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:35.737077 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:35.738425 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:35.739165 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:35.740772 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:35.741193 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:35.741324 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:35.744532 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:35.744538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:35.744541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:35.744572 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:35.747150 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:35.748012 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:35.748020 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:35.748029 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:35.748147 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:35.748617 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:35.748621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:35.748623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:35.748975 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:35.749139 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:35.749797 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 03:08:35.750016 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":292,"threads_started":1}
I20260620 03:08:35.751448 13235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:35.733555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41153 (local address 127.0.97.3:41861)
0620 03:08:35.734458 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:35.734508 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:35.736641 (+ 2133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:35.737276 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:35.737284 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:35.737705 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:35.738257 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:35.738273 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:35.739359 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:35.739365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:35.740634 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:35.740645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:35.741394 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:35.741399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:35.741610 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:35.742229 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:35.742254 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:35.744294 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:35.747319 (+ 3025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:35.747331 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:35.747349 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:35.747893 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:35.748264 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:35.748268 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:35.748272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:35.748472 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:35.749099 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:35.749107 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:35.749453 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 03:08:35.749712 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 03:08:35.750222 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":380,"threads_started":1}
I20260620 03:08:35.752751 13232 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41153
I20260620 03:08:35.753032 13232 heartbeater.cc:464] Registering TS with master...
I20260620 03:08:35.753750 13232 heartbeater.cc:511] Master 127.0.97.62:41153 requested a full tablet report, sending...
I20260620 03:08:35.755036 12742 ts_manager.cc:194] Registered new tserver with Master: 303a3ced4f8541ae9bcc3b25611bec62 (127.0.97.3:45325)
I20260620 03:08:35.755883 12742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41861
I20260620 03:08:35.761741 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:08:35.763459 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 12822
Jun 20 03:08:35 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:08:35.780351 13237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:35.764562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.0.97.1:37675)
0620 03:08:35.765223 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:35.765229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:35.765256 (+ 27us) server_negotiation.cc:407] Connection header received
0620 03:08:35.765320 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:35.765326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:35.765604 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:35.765741 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:35.767476 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:35.768517 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:35.769977 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:35.770559 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:35.770662 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:35.774984 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:35.774998 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:35.775013 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:35.775051 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:35.777936 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:35.778592 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:35.778599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:35.778601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:35.778737 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:35.779029 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:35.779032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:35.779033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:35.779410 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:35.779638 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:35.779876 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 03:08:35.780075 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":318,"threads_started":1}
I20260620 03:08:35.781590 12950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:35.864004 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 12822
I20260620 03:08:35.879685 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 12961
Jun 20 03:08:35 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:08:35.896726 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:35.880604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.0.97.2:35187)
0620 03:08:35.881173 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:35.881181 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:35.881207 (+ 26us) server_negotiation.cc:407] Connection header received
0620 03:08:35.881269 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:35.881275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:35.881490 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:35.881616 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:35.882863 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:35.883951 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:35.885508 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:35.886082 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:35.886186 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:35.890675 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:35.890688 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:35.890703 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:35.890747 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:35.893644 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:35.894399 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:35.894407 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:35.894409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:35.894557 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:35.894934 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:35.894940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:35.894943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:35.895443 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:35.895747 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:35.896080 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 03:08:35.896350 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":296,"threads_started":1}
I20260620 03:08:35.898022 13093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:36.002970 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 12961
I20260620 03:08:36.019143 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 13104
Jun 20 03:08:36 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:08:36.035357 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:36.020275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.0.97.3:45325)
0620 03:08:36.020900 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:36.020908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:36.020936 (+ 28us) server_negotiation.cc:407] Connection header received
0620 03:08:36.021002 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:36.021008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:36.021210 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:36.021386 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:36.022653 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:36.023749 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:36.024983 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:36.025532 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:36.025665 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:36.029965 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:36.029975 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:36.029988 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:36.030019 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:36.032655 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:36.033297 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:36.033304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:36.033306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:36.033428 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:36.033773 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:36.033780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:36.033783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:36.034100 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:36.034407 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:36.034701 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 03:08:36.034960 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":334,"threads_started":1}
I20260620 03:08:36.036629 13227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:36.118588 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 13104
I20260620 03:08:36.134658 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 12710
I20260620 03:08:36.135977 12772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:36.241788 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 12710
I20260620 03:08:36.266283 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41153
--webserver_interface=127.0.97.62
--webserver_port=45973
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41153
--ranger_config_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45741/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45741/kms with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:08:36.590396 13244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:36.590760 13244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:36.590931 13244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:36.598520 13244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:08:36.598699 13244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:36.598811 13244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:36.598908 13244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:08:36.598999 13244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:08:36.609129 13244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45741/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41153
--ranger_config_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:36.610491 13244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:36.611994 13244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:36.623660 13249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:36.623622 13250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:36.623565 13252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:36.625203 13244 server_base.cc:1034] running on GCE node
I20260620 03:08:36.625929 13244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:36.627390 13244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:36.628618 13244 hybrid_clock.cc:648] HybridClock initialized: now 1781924916628578 us; error 36 us; skew 500 ppm
I20260620 03:08:36.628957 13244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:36 dist-test-slave-xv68 krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:36.636662 13244 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:08:36.638125 13244 webserver.cc:460] Webserver started at http://127.0.97.62:45973/ using document root <none> and password file <none>
I20260620 03:08:36.638721 13244 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:36.638823 13244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 03:08:36 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:08:36.723143 13244 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 03:08:36.726269 13260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:36.727900 13244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:08:36.728124 13244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "45885db77bf8404fb8a6dc5535f6f794"
format_stamp: "Formatted at 2026-06-20 03:08:32 on dist-test-slave-xv68"
server_key: "36bdca808b2ddfbcc2226c57c5805291"
server_key_iv: "dfc8a7a08e278d4605b0cb3905b2e003"
server_key_version: "kuduclusterkey@0"
I20260620 03:08:36.729144 13244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:36.741845 13244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:36.745556 13244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:36.745836 13244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:36.763950 13244 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41153
I20260620 03:08:36.763996 13311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41153 every 8 connection(s)
I20260620 03:08:36.765401 13244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:08:36.770344 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 13244
I20260620 03:08:36.771430 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45741/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37675
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41153
--webserver_port=43007
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45741/kms with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:08:36.773144 13312 sys_catalog.cc:263] Verifying existing consensus state
I20260620 03:08:36.778487 13312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794: Bootstrap starting.
I20260620 03:08:36.797746 13312 log.cc:824] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:36.806517 13312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 03:08:36.807062 13312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794: Bootstrap complete.
I20260620 03:08:36.816169 13312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45885db77bf8404fb8a6dc5535f6f794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41153 } }
I20260620 03:08:36.816978 13312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45885db77bf8404fb8a6dc5535f6f794, State: Initialized, Role: FOLLOWER
I20260620 03:08:36.817731 13312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45885db77bf8404fb8a6dc5535f6f794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41153 } }
I20260620 03:08:36.817996 13312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:08:36.818166 13312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:08:36.818357 13312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [term 1 FOLLOWER]: Advancing to term 2
I20260620 03:08:36.822039 13312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45885db77bf8404fb8a6dc5535f6f794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41153 } }
I20260620 03:08:36.822520 13312 leader_election.cc:302] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45885db77bf8404fb8a6dc5535f6f794; no voters:
I20260620 03:08:36.823168 13312 leader_election.cc:288] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 03:08:36.823359 13316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 03:08:36.824352 13316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [term 2 LEADER]: Becoming Leader. State: Replica: 45885db77bf8404fb8a6dc5535f6f794, State: Running, Role: LEADER
I20260620 03:08:36.824972 13316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45885db77bf8404fb8a6dc5535f6f794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41153 } }
I20260620 03:08:36.825330 13312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:08:36.827358 13318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45885db77bf8404fb8a6dc5535f6f794. Latest consensus state: current_term: 2 leader_uuid: "45885db77bf8404fb8a6dc5535f6f794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45885db77bf8404fb8a6dc5535f6f794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41153 } } }
I20260620 03:08:36.827684 13318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [sys.catalog]: This master's current role is: LEADER
I20260620 03:08:36.827504 13317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "45885db77bf8404fb8a6dc5535f6f794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45885db77bf8404fb8a6dc5535f6f794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41153 } } }
I20260620 03:08:36.828047 13317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794 [sys.catalog]: This master's current role is: LEADER
I20260620 03:08:36.842402 13323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:08:36.848343 13323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:08:36.849735 13323 catalog_manager.cc:1259] Loaded cluster ID: 794e62162aab4a25b041b2a60863a144
I20260620 03:08:36.849901 13323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:08:36.853826 13323 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:08:36.855239 13323 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794: Loaded TSK: 0
I20260620 03:08:36.856858 13323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:08:36.859400 13312 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 03:08:37.174703 13314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:37.175200 13314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:37.175472 13314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:37.187480 13314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:37.187736 13314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:37.187986 13314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:08:37.204396 13314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45741/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43007
--tserver_master_addrs=127.0.97.62:41153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:37.206416 13314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:37.208271 13314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:37.234319 13350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:37.234615 13349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:37.237382 13353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:37.237908 13314 server_base.cc:1034] running on GCE node
I20260620 03:08:37.238840 13314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:37.243854 13314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:37.245170 13314 hybrid_clock.cc:648] HybridClock initialized: now 1781924917245059 us; error 109 us; skew 500 ppm
I20260620 03:08:37.245627 13314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:37 dist-test-slave-xv68 krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:37.250070 13314 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:08:37.269634 13314 webserver.cc:460] Webserver started at http://127.0.97.1:43007/ using document root <none> and password file <none>
I20260620 03:08:37.270717 13314 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:37.270977 13314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 03:08:37 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:08:37.314112 13314 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.006s sys 0.007s
I20260620 03:08:37.331233 13360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:37.337441 13314 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.009s sys 0.006s
I20260620 03:08:37.343175 13314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "e360b58de2934d0e90cf908159b0b5ff"
format_stamp: "Formatted at 2026-06-20 03:08:34 on dist-test-slave-xv68"
server_key: "8b4f2cfe58fec582a7fc4ca3268302d6"
server_key_iv: "0d94172b9962218830f12bcec93a9ef0"
server_key_version: "kuduclusterkey@0"
I20260620 03:08:37.347806 13314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:37.427919 13314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:37.432864 13314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:37.433163 13314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:37.443856 13314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:08:37.447225 13314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:08:37.447441 13314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 03:08:37.447574 13314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:08:37.447700 13314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:37.511112 13314 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37675
I20260620 03:08:37.511415 13473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37675 every 8 connection(s)
I20260620 03:08:37.512878 13314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:08:37.514758 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 13314
I20260620 03:08:37.515648 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45741/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35187
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41153
--webserver_port=40311
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45741/kms with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:08:37 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:08:37.560739 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:37.516419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41153 (local address 127.0.97.1:44205)
0620 03:08:37.535559 (+ 19140us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:37.535627 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:37.538146 (+ 2519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:37.540702 (+ 2556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:37.540713 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:37.541350 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:37.542111 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:37.542136 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:37.544600 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:37.544605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:37.546643 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:37.546658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:37.548451 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:37.548457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:37.548853 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:37.549616 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:37.549652 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:37.552247 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:37.556134 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:37.556144 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:37.556159 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:37.556609 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:37.557056 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:37.557061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:37.557064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:37.557259 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:37.558176 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:37.558186 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:37.558475 (+ 289us) client_negotiation.cc:769] Sending connection context
0620 03:08:37.558750 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 03:08:37.559443 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18719,"thread_start_us":22058,"threads_started":1}
I20260620 03:08:37.560739 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:37.516992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44205 (local address 127.0.97.62:41153)
0620 03:08:37.531858 (+ 14866us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:37.531882 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:37.535794 (+ 3912us) server_negotiation.cc:407] Connection header received
0620 03:08:37.539027 (+ 3233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:37.539046 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:37.539601 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:37.540027 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:37.542337 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:37.544356 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:37.546820 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:37.547877 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:37.548018 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:37.552571 (+ 4553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:37.552585 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:37.552604 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:37.552654 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:37.555984 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:37.556759 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:37.556771 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:37.556783 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:37.556933 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:37.557380 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:37.557387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:37.557390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:37.557909 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:37.558087 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:37.558950 (+ 863us) server_negotiation.cc:299] Negotiation successful
0620 03:08:37.559359 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14541,"thread_start_us":14340,"threads_started":1}
I20260620 03:08:37.564766 13474 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41153
I20260620 03:08:37.565094 13474 heartbeater.cc:464] Registering TS with master...
I20260620 03:08:37.565809 13474 heartbeater.cc:511] Master 127.0.97.62:41153 requested a full tablet report, sending...
W20260620 03:08:37.568908 13474 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41153 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 03:08:37.569147 13474 heartbeater.cc:464] Registering TS with master...
I20260620 03:08:37.569418 13474 heartbeater.cc:511] Master 127.0.97.62:41153 requested a full tablet report, sending...
I20260620 03:08:37.570837 13474 heartbeater.cc:464] Registering TS with master...
I20260620 03:08:37.571102 13474 heartbeater.cc:511] Master 127.0.97.62:41153 requested a full tablet report, sending...
W20260620 03:08:37.572273 13474 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 03:08:37.949679 13478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:37.950083 13478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:37.950297 13478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:37.962013 13478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:37.962172 13478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:37.962419 13478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:08:37.974221 13478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45741/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40311
--tserver_master_addrs=127.0.97.62:41153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:37.975746 13478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:37.977069 13478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:38.014442 13492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:38.015971 13493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:38.037626 13478 server_base.cc:1034] running on GCE node
W20260620 03:08:38.038683 13497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:38.039614 13478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:38.041510 13478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:38.042804 13478 hybrid_clock.cc:648] HybridClock initialized: now 1781924918042743 us; error 50 us; skew 500 ppm
I20260620 03:08:38.043205 13478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:38 dist-test-slave-xv68 krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:38.047461 13478 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:08:38.049505 13478 webserver.cc:460] Webserver started at http://127.0.97.2:40311/ using document root <none> and password file <none>
I20260620 03:08:38.050413 13478 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:38.050566 13478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 03:08:38 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:08:38.086215 13478 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 03:08:38.090106 13505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:38.091784 13478 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260620 03:08:38.091997 13478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "2c831e7bb0c440299b2482eb8695facf"
format_stamp: "Formatted at 2026-06-20 03:08:35 on dist-test-slave-xv68"
server_key: "3fa64d9fe4395f5cf8c5825cc763dd50"
server_key_iv: "4009f9e80fd880c17458cb156caef8be"
server_key_version: "kuduclusterkey@0"
I20260620 03:08:38.093139 13478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 20 03:08:38 dist-test-slave-xv68 krb5kdc[12266](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781924918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:38.131829 13478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:38.136772 13478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:38.137060 13478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:38.138132 13478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:08:38.139792 13478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:08:38.139896 13478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:38.139999 13478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:08:38.140069 13478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:38.211856 13478 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35187
I20260620 03:08:38.211936 13618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35187 every 8 connection(s)
I20260620 03:08:38.213829 13478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:08:38.218410 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 13478
I20260620 03:08:38.219178 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45741/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45325
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41153
--webserver_port=44801
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45741/kms with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:08:38 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:08:38.245100 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:38.221878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33289 (local address 127.0.97.62:41153)
0620 03:08:38.223871 (+ 1993us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:38.223877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:38.223897 (+ 20us) server_negotiation.cc:407] Connection header received
0620 03:08:38.223967 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:38.223972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:38.224056 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:38.224170 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:38.228288 (+ 4118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:38.229473 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:38.231970 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:38.232656 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:38.232784 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:38.237441 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:38.237448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:38.237455 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:38.237510 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:38.241111 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:38.242091 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:38.242097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:38.242100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:38.242277 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:38.242877 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:38.242882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:38.242885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:38.243247 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:38.243452 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:38.244479 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0620 03:08:38.244719 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1867,"thread_start_us":3756,"threads_started":1}
I20260620 03:08:38.247789 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:38.217501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41153 (local address 127.0.97.2:33289)
0620 03:08:38.219068 (+ 1567us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:38.219129 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:38.222598 (+ 3469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:38.226467 (+ 3869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:38.226481 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:38.227183 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:38.228070 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:38.228096 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:38.229715 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:38.229720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:38.231779 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:38.231807 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:38.233123 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:38.233130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:38.233401 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:38.234265 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:38.234296 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:38.237161 (+ 2865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:38.241282 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:38.241293 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:38.241312 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:38.241939 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:38.242433 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:38.242440 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:38.242444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:38.242731 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:38.243566 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:38.243576 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:38.244026 (+ 450us) client_negotiation.cc:769] Sending connection context
0620 03:08:38.245655 (+ 1629us) client_negotiation.cc:240] Negotiation successful
0620 03:08:38.246246 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1254,"thread_start_us":4404,"threads_started":1}
I20260620 03:08:38.249889 13619 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41153
I20260620 03:08:38.250211 13619 heartbeater.cc:464] Registering TS with master...
I20260620 03:08:38.251132 13619 heartbeater.cc:511] Master 127.0.97.62:41153 requested a full tablet report, sending...
I20260620 03:08:38.253369 13277 ts_manager.cc:194] Registered new tserver with Master: 2c831e7bb0c440299b2482eb8695facf (127.0.97.2:35187)
I20260620 03:08:38.255842 13277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33289
Jun 20 03:08:38 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:08:38.572424 13474 heartbeater.cc:464] Registering TS with master...
I20260620 03:08:38.572891 13474 heartbeater.cc:511] Master 127.0.97.62:41153 requested a full tablet report, sending...
I20260620 03:08:38.574263 13277 ts_manager.cc:194] Registered new tserver with Master: e360b58de2934d0e90cf908159b0b5ff (127.0.97.1:37675)
I20260620 03:08:38.575584 13277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44205
W20260620 03:08:38.695160 13623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:38.695701 13623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:38.695885 13623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:38.706454 13623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:38.706683 13623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:38.706884 13623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:08:38.720893 13623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45741/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44801
--tserver_master_addrs=127.0.97.62:41153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:38.722848 13623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:38.724632 13623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:38.741966 13628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:38.741945 13629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:38.748775 13631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:38.749670 13623 server_base.cc:1034] running on GCE node
I20260620 03:08:38.750416 13623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:38.751873 13623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:38.753154 13623 hybrid_clock.cc:648] HybridClock initialized: now 1781924918753089 us; error 52 us; skew 500 ppm
I20260620 03:08:38.753523 13623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:38 dist-test-slave-xv68 krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:38.757653 13623 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:08:38.759588 13623 webserver.cc:460] Webserver started at http://127.0.97.3:44801/ using document root <none> and password file <none>
I20260620 03:08:38.760558 13623 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:38.760718 13623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 03:08:38 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:08:38.798136 13623 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.004s
I20260620 03:08:38.802078 13639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:38.803790 13623 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260620 03:08:38.804041 13623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "303a3ced4f8541ae9bcc3b25611bec62"
format_stamp: "Formatted at 2026-06-20 03:08:35 on dist-test-slave-xv68"
server_key: "56a1d5ef53355430307cc7b45ad7a299"
server_key_iv: "0b4a6670c9b79f0532c0ea53597e68b9"
server_key_version: "kuduclusterkey@0"
I20260620 03:08:38.805496 13623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:38.839627 13623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Jun 20 03:08:38 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:08:38.846328 13623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:38.846664 13623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:38.847886 13623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:08:38.849711 13623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:08:38.849819 13623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:38.849923 13623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:08:38.850004 13623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:38.988559 13623 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45325
I20260620 03:08:38.988629 13752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45325 every 8 connection(s)
I20260620 03:08:38.990532 13623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:08:38.996466 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 13623
Jun 20 03:08:39 dist-test-slave-xv68 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:08:39.035425 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:38.997827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42766 (local address 127.0.97.62:41153)
0620 03:08:39.008969 (+ 11142us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:39.008987 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:39.011815 (+ 2828us) server_negotiation.cc:407] Connection header received
0620 03:08:39.012422 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:39.012431 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:39.012533 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:39.012648 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:39.016638 (+ 3990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.017777 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.019854 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.020473 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.020586 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:39.024595 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:39.024602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:39.024608 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:39.024646 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:39.028660 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:39.029385 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:39.029392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:39.029395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:39.029553 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:39.032370 (+ 2817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:39.032377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:39.032381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:39.032775 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:39.033024 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:39.034758 (+ 1734us) server_negotiation.cc:299] Negotiation successful
0620 03:08:39.034982 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10956,"thread_start_us":11160,"threads_started":1}
I20260620 03:08:39.041100 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.009192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59793 (local address 127.0.97.62:41153)
0620 03:08:39.016454 (+ 7262us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:39.016460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:39.016478 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:08:39.018411 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:39.018418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:39.018504 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:39.018601 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:39.020711 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.021951 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.024537 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.025258 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.025376 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:39.030589 (+ 5213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:39.030597 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:39.030602 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:39.030638 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:39.034541 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:39.038351 (+ 3810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:39.038357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:39.038361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:39.038517 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:39.039067 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:39.039072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:39.039075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:39.039461 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:39.039621 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:39.040519 (+ 898us) server_negotiation.cc:299] Negotiation successful
0620 03:08:39.040736 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7142,"thread_start_us":7275,"threads_started":1}
I20260620 03:08:39.042418 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.000671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41153 (local address 127.0.97.3:59793)
0620 03:08:39.013297 (+ 12626us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:39.013368 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:39.017817 (+ 4449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:39.019045 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:39.019057 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:39.019694 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:39.020483 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:39.020508 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.022167 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.022173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:39.024362 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:39.024375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.025762 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.025771 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:39.026075 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:39.027517 (+ 1442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:39.027546 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:39.030308 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:39.036364 (+ 6056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:39.036373 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:39.036392 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:39.038201 (+ 1809us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:39.038657 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:39.038663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:39.038666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:39.038929 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:39.039603 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:39.039611 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:39.040053 (+ 442us) client_negotiation.cc:769] Sending connection context
0620 03:08:39.040365 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 03:08:39.040935 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12264,"thread_start_us":15204,"threads_started":1}
I20260620 03:08:39.044406 13753 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41153
I20260620 03:08:39.044747 13753 heartbeater.cc:464] Registering TS with master...
I20260620 03:08:39.045572 13753 heartbeater.cc:511] Master 127.0.97.62:41153 requested a full tablet report, sending...
I20260620 03:08:39.047223 13277 ts_manager.cc:194] Registered new tserver with Master: 303a3ced4f8541ae9bcc3b25611bec62 (127.0.97.3:45325)
I20260620 03:08:39.048359 13277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59793
I20260620 03:08:39.052711 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:08:39.075845 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.056189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.0.97.62:41153)
0620 03:08:39.056363 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:39.056369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:39.059866 (+ 3497us) server_negotiation.cc:407] Connection header received
0620 03:08:39.060016 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:39.060021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:39.060118 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:39.060226 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:39.062239 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.063367 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.065187 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.065820 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.065956 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:39.069205 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:39.069213 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:39.069218 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:39.069260 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:39.072775 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:39.073658 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:39.073667 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:39.073670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:39.073818 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:39.074239 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:39.074245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:39.074248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:39.074630 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:39.074874 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:39.075235 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 03:08:39.075516 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:08:39.081996 13277 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:08:39.093024 13277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:08:39.168605 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.121024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60014 (local address 127.0.97.3:45325)
0620 03:08:39.155255 (+ 34231us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:39.155269 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:39.159194 (+ 3925us) server_negotiation.cc:407] Connection header received
0620 03:08:39.159562 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:39.159571 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:39.159801 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:39.159993 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:08:39.160890 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.162490 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.165218 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.166620 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.166731 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:39.166949 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:39.168089 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0620 03:08:39.168254 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33981,"thread_start_us":35205,"threads_started":1}
I20260620 03:08:39.169178 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.118303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35187 (local address 127.0.0.1:40656)
0620 03:08:39.154013 (+ 35710us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:39.154083 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:39.154408 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:39.159847 (+ 5439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:39.159856 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:39.159891 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:08:39.160362 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:39.160399 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.164075 (+ 3676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.164081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:39.166266 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:39.166280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.168231 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.168237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:39.168429 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:39.168475 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 03:08:39.168543 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 03:08:39.168684 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35290,"mutex_wait_us":3,"thread_start_us":40420,"threads_started":1}
I20260620 03:08:39.169425 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.120403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45325 (local address 127.0.0.1:60014)
0620 03:08:39.159066 (+ 38663us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:39.159089 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:39.159318 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:39.160276 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:39.160280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:39.160313 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:08:39.160686 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:39.160697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.162674 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.162680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:39.165042 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:39.165057 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.167135 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.167141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:39.167363 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:39.167413 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 03:08:39.167547 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 03:08:39.169105 (+ 1558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38492,"thread_start_us":34971,"threads_started":1}
I20260620 03:08:39.172263 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.118618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37675 (local address 127.0.0.1:42812)
0620 03:08:39.156964 (+ 38346us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:39.156988 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:39.157142 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:39.163787 (+ 6645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:39.163792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:39.163820 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:08:39.164267 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:39.164284 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.167552 (+ 3268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.167557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:39.169635 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:39.169649 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.171529 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.171539 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:39.171707 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:39.171736 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 03:08:39.171802 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 03:08:39.171945 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38185,"thread_start_us":38623,"threads_started":1}
I20260620 03:08:39.172415 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.118888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.0.97.2:35187)
0620 03:08:39.157972 (+ 39084us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:39.157985 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:39.158025 (+ 40us) server_negotiation.cc:407] Connection header received
0620 03:08:39.158160 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:39.158172 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:39.159575 (+ 1403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:39.160654 (+ 1079us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:08:39.160935 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.162658 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.166488 (+ 3830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.167988 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.171608 (+ 3620us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:39.171820 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:39.171895 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 03:08:39.172067 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38891,"thread_start_us":38752,"threads_started":1}
I20260620 03:08:39.173817 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.118634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42812 (local address 127.0.97.1:37675)
0620 03:08:39.163165 (+ 44531us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:39.163175 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:39.163202 (+ 27us) server_negotiation.cc:407] Connection header received
0620 03:08:39.163327 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:39.163336 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:39.163593 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:39.163795 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:08:39.164502 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.167393 (+ 2891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.169839 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.171255 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.172964 (+ 1709us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:39.173183 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:39.173250 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 03:08:39.173416 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44363,"thread_start_us":44699,"threads_started":1}
I20260620 03:08:39.173785 13688 tablet_service.cc:1467] Processing CreateTablet for tablet 5fdab43e13714ca6aaafff998a6d47ab (DEFAULT_TABLE table=test-table [id=1c6d5a663f3d40d197ad99bc05fb66bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:08:39.175381 13688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fdab43e13714ca6aaafff998a6d47ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:08:39.176167 13553 tablet_service.cc:1467] Processing CreateTablet for tablet 5fdab43e13714ca6aaafff998a6d47ab (DEFAULT_TABLE table=test-table [id=1c6d5a663f3d40d197ad99bc05fb66bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:08:39.177508 13553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fdab43e13714ca6aaafff998a6d47ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:08:39.177341 13409 tablet_service.cc:1467] Processing CreateTablet for tablet 5fdab43e13714ca6aaafff998a6d47ab (DEFAULT_TABLE table=test-table [id=1c6d5a663f3d40d197ad99bc05fb66bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:08:39.178584 13409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fdab43e13714ca6aaafff998a6d47ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:08:39.200671 13782 tablet_bootstrap.cc:492] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff: Bootstrap starting.
I20260620 03:08:39.204051 13781 tablet_bootstrap.cc:492] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62: Bootstrap starting.
I20260620 03:08:39.207161 13781 tablet_bootstrap.cc:654] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62: Neither blocks nor log segments found. Creating new log.
I20260620 03:08:39.207947 13782 tablet_bootstrap.cc:654] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff: Neither blocks nor log segments found. Creating new log.
I20260620 03:08:39.208493 13781 log.cc:824] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:39.208842 13783 tablet_bootstrap.cc:492] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf: Bootstrap starting.
I20260620 03:08:39.213058 13782 log.cc:824] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:39.215341 13781 tablet_bootstrap.cc:492] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62: No bootstrap required, opened a new log
I20260620 03:08:39.215687 13781 ts_tablet_manager.cc:1397] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260620 03:08:39.216684 13783 tablet_bootstrap.cc:654] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf: Neither blocks nor log segments found. Creating new log.
I20260620 03:08:39.218555 13783 log.cc:824] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:39.228228 13781 raft_consensus.cc:348] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c831e7bb0c440299b2482eb8695facf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35187 } } peers { permanent_uuid: "e360b58de2934d0e90cf908159b0b5ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37675 } } peers { permanent_uuid: "303a3ced4f8541ae9bcc3b25611bec62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45325 } }
I20260620 03:08:39.228647 13781 raft_consensus.cc:374] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:08:39.228770 13781 raft_consensus.cc:729] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303a3ced4f8541ae9bcc3b25611bec62, State: Initialized, Role: FOLLOWER
I20260620 03:08:39.229401 13781 consensus_queue.cc:260] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c831e7bb0c440299b2482eb8695facf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35187 } } peers { permanent_uuid: "e360b58de2934d0e90cf908159b0b5ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37675 } } peers { permanent_uuid: "303a3ced4f8541ae9bcc3b25611bec62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45325 } }
I20260620 03:08:39.229806 13782 tablet_bootstrap.cc:492] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff: No bootstrap required, opened a new log
I20260620 03:08:39.230147 13782 ts_tablet_manager.cc:1397] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.005s
I20260620 03:08:39.230460 13753 heartbeater.cc:503] Master 127.0.97.62:41153 was elected leader, sending a full tablet report...
I20260620 03:08:39.231146 13781 ts_tablet_manager.cc:1428] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62: Time spent starting tablet: real 0.015s user 0.009s sys 0.003s
I20260620 03:08:39.234037 13783 tablet_bootstrap.cc:492] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf: No bootstrap required, opened a new log
I20260620 03:08:39.234338 13783 ts_tablet_manager.cc:1397] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.005s
I20260620 03:08:39.240245 13782 raft_consensus.cc:348] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c831e7bb0c440299b2482eb8695facf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35187 } } peers { permanent_uuid: "e360b58de2934d0e90cf908159b0b5ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37675 } } peers { permanent_uuid: "303a3ced4f8541ae9bcc3b25611bec62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45325 } }
I20260620 03:08:39.240674 13782 raft_consensus.cc:374] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:08:39.240763 13782 raft_consensus.cc:729] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e360b58de2934d0e90cf908159b0b5ff, State: Initialized, Role: FOLLOWER
I20260620 03:08:39.241356 13782 consensus_queue.cc:260] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c831e7bb0c440299b2482eb8695facf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35187 } } peers { permanent_uuid: "e360b58de2934d0e90cf908159b0b5ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37675 } } peers { permanent_uuid: "303a3ced4f8541ae9bcc3b25611bec62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45325 } }
I20260620 03:08:39.242408 13474 heartbeater.cc:503] Master 127.0.97.62:41153 was elected leader, sending a full tablet report...
I20260620 03:08:39.243024 13782 ts_tablet_manager.cc:1428] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 03:08:39.243798 13783 raft_consensus.cc:348] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c831e7bb0c440299b2482eb8695facf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35187 } } peers { permanent_uuid: "e360b58de2934d0e90cf908159b0b5ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37675 } } peers { permanent_uuid: "303a3ced4f8541ae9bcc3b25611bec62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45325 } }
I20260620 03:08:39.244259 13783 raft_consensus.cc:374] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:08:39.244379 13783 raft_consensus.cc:729] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c831e7bb0c440299b2482eb8695facf, State: Initialized, Role: FOLLOWER
I20260620 03:08:39.245003 13783 consensus_queue.cc:260] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c831e7bb0c440299b2482eb8695facf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35187 } } peers { permanent_uuid: "e360b58de2934d0e90cf908159b0b5ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37675 } } peers { permanent_uuid: "303a3ced4f8541ae9bcc3b25611bec62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45325 } }
I20260620 03:08:39.246244 13619 heartbeater.cc:503] Master 127.0.97.62:41153 was elected leader, sending a full tablet report...
I20260620 03:08:39.246711 13783 ts_tablet_manager.cc:1428] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
W20260620 03:08:39.265724 13475 tablet.cc:2369] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:08:39.266289 13757 tablet.cc:2369] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:08:39.350658 13789 raft_consensus.cc:482] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:08:39.350975 13789 raft_consensus.cc:504] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c831e7bb0c440299b2482eb8695facf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35187 } } peers { permanent_uuid: "e360b58de2934d0e90cf908159b0b5ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37675 } } peers { permanent_uuid: "303a3ced4f8541ae9bcc3b25611bec62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45325 } }
I20260620 03:08:39.352845 13789 leader_election.cc:288] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e360b58de2934d0e90cf908159b0b5ff (127.0.97.1:37675), 303a3ced4f8541ae9bcc3b25611bec62 (127.0.97.3:45325)
I20260620 03:08:39.360414 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.353532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32893 (local address 127.0.97.1:37675)
0620 03:08:39.353725 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:39.353731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:39.354511 (+ 780us) server_negotiation.cc:407] Connection header received
0620 03:08:39.354793 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:39.354798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:39.354887 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:39.354993 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:08:39.355641 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.356443 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.358634 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.359466 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.359623 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:39.359736 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:39.360013 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 03:08:39.360139 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:08:39.360456 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.352982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37675 (local address 127.0.97.2:32893)
0620 03:08:39.354395 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:39.354418 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:39.354648 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:39.355025 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:39.355030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:39.355054 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:08:39.355469 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:39.355483 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.356596 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.356599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:39.358475 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:39.358488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.359698 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.359704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:39.359855 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:39.359879 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 03:08:39.359997 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 03:08:39.360127 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1189,"thread_start_us":930,"threads_started":1}
I20260620 03:08:39.361743 13429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdab43e13714ca6aaafff998a6d47ab" candidate_uuid: "2c831e7bb0c440299b2482eb8695facf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e360b58de2934d0e90cf908159b0b5ff" is_pre_election: true
I20260620 03:08:39.362329 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.353530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35883 (local address 127.0.97.3:45325)
0620 03:08:39.353752 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:39.353757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:39.355431 (+ 1674us) server_negotiation.cc:407] Connection header received
0620 03:08:39.355813 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:39.355819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:39.355892 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:39.355951 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:08:39.356743 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.357800 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.360258 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.361411 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.361532 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:39.361622 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:39.361975 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 03:08:39.362107 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 03:08:39.362365 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.353252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45325 (local address 127.0.97.2:35883)
0620 03:08:39.355307 (+ 2055us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:39.355352 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:39.355530 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:39.356143 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:39.356148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:39.356172 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:08:39.356533 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:39.356549 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.357961 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.357967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:39.360078 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:39.360092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.361619 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.361628 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:39.361798 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:39.361830 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 03:08:39.361932 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 03:08:39.362071 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1184,"thread_start_us":987,"threads_started":1}
I20260620 03:08:39.362330 13429 raft_consensus.cc:2454] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c831e7bb0c440299b2482eb8695facf in term 0.
I20260620 03:08:39.363076 13507 leader_election.cc:302] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c831e7bb0c440299b2482eb8695facf, e360b58de2934d0e90cf908159b0b5ff; no voters:
I20260620 03:08:39.363488 13789 raft_consensus.cc:2781] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:08:39.363444 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdab43e13714ca6aaafff998a6d47ab" candidate_uuid: "2c831e7bb0c440299b2482eb8695facf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303a3ced4f8541ae9bcc3b25611bec62" is_pre_election: true
I20260620 03:08:39.363626 13789 raft_consensus.cc:482] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:08:39.363718 13789 raft_consensus.cc:3037] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:08:39.363935 13708 raft_consensus.cc:2454] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c831e7bb0c440299b2482eb8695facf in term 0.
I20260620 03:08:39.365921 13789 raft_consensus.cc:504] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c831e7bb0c440299b2482eb8695facf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35187 } } peers { permanent_uuid: "e360b58de2934d0e90cf908159b0b5ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37675 } } peers { permanent_uuid: "303a3ced4f8541ae9bcc3b25611bec62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45325 } }
I20260620 03:08:39.366550 13789 leader_election.cc:288] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [CANDIDATE]: Term 1 election: Requested vote from peers e360b58de2934d0e90cf908159b0b5ff (127.0.97.1:37675), 303a3ced4f8541ae9bcc3b25611bec62 (127.0.97.3:45325)
I20260620 03:08:39.366967 13429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdab43e13714ca6aaafff998a6d47ab" candidate_uuid: "2c831e7bb0c440299b2482eb8695facf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e360b58de2934d0e90cf908159b0b5ff"
I20260620 03:08:39.367049 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdab43e13714ca6aaafff998a6d47ab" candidate_uuid: "2c831e7bb0c440299b2482eb8695facf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303a3ced4f8541ae9bcc3b25611bec62"
I20260620 03:08:39.367187 13429 raft_consensus.cc:3037] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:08:39.367223 13708 raft_consensus.cc:3037] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:08:39.370368 13429 raft_consensus.cc:2454] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c831e7bb0c440299b2482eb8695facf in term 1.
I20260620 03:08:39.370368 13708 raft_consensus.cc:2454] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c831e7bb0c440299b2482eb8695facf in term 1.
I20260620 03:08:39.370755 13507 leader_election.cc:302] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c831e7bb0c440299b2482eb8695facf, e360b58de2934d0e90cf908159b0b5ff; no voters:
I20260620 03:08:39.371074 13789 raft_consensus.cc:2781] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:08:39.371693 13789 raft_consensus.cc:686] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [term 1 LEADER]: Becoming Leader. State: Replica: 2c831e7bb0c440299b2482eb8695facf, State: Running, Role: LEADER
I20260620 03:08:39.372162 13789 consensus_queue.cc:237] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c831e7bb0c440299b2482eb8695facf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35187 } } peers { permanent_uuid: "e360b58de2934d0e90cf908159b0b5ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37675 } } peers { permanent_uuid: "303a3ced4f8541ae9bcc3b25611bec62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45325 } }
I20260620 03:08:39.376425 13277 catalog_manager.cc:5495] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c831e7bb0c440299b2482eb8695facf (127.0.97.2). New cstate: current_term: 1 leader_uuid: "2c831e7bb0c440299b2482eb8695facf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c831e7bb0c440299b2482eb8695facf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e360b58de2934d0e90cf908159b0b5ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "303a3ced4f8541ae9bcc3b25611bec62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45325 } health_report { overall_health: UNKNOWN } } }
I20260620 03:08:39.416486 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.410124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40658 (local address 127.0.97.2:35187)
0620 03:08:39.410327 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:39.410332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:39.410350 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:08:39.410619 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:39.410623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:39.410711 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:39.410873 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:08:39.411239 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.412128 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.414110 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.414506 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.414599 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:39.415162 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:08:39.415744 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:08:39.415850 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:39.416105 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 03:08:39.416169 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 03:08:39.424626 13429 raft_consensus.cc:1261] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff [term 1 FOLLOWER]: Refusing update from remote peer 2c831e7bb0c440299b2482eb8695facf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:08:39.424737 13708 raft_consensus.cc:1261] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62 [term 1 FOLLOWER]: Refusing update from remote peer 2c831e7bb0c440299b2482eb8695facf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:08:39.425729 13792 consensus_queue.cc:1035] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [LEADER]: Connected to new peer: Peer: permanent_uuid: "303a3ced4f8541ae9bcc3b25611bec62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:08:39.426117 13789 consensus_queue.cc:1035] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e360b58de2934d0e90cf908159b0b5ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:08:39.443004 13796 mvcc.cc:205] Tried to move back new op lower bound from 7298764469952851968 to 7298764469751934976. Current Snapshot: MvccSnapshot[applied={T|T < 7298764469952851968}]
I20260620 03:08:39.443316 13795 mvcc.cc:205] Tried to move back new op lower bound from 7298764469952851968 to 7298764469751934976. Current Snapshot: MvccSnapshot[applied={T|T < 7298764469952851968}]
I20260620 03:08:39.467813 13277 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:08:39.468103 13277 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:08:39.472090 13277 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45885db77bf8404fb8a6dc5535f6f794: Sending DeleteTablet for 3 replicas of tablet 5fdab43e13714ca6aaafff998a6d47ab
I20260620 03:08:39.473649 13553 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fdab43e13714ca6aaafff998a6d47ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:08:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40656
I20260620 03:08:39.474344 13409 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fdab43e13714ca6aaafff998a6d47ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:08:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42812
I20260620 03:08:39.474546 13805 tablet_replica.cc:330] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf: stopping tablet replica
I20260620 03:08:39.474957 13805 raft_consensus.cc:2229] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:08:39.475461 13688 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fdab43e13714ca6aaafff998a6d47ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:08:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60014
I20260620 03:08:39.475716 13805 raft_consensus.cc:2258] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:08:39.476387 13807 tablet_replica.cc:330] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62: stopping tablet replica
I20260620 03:08:39.476935 13807 raft_consensus.cc:2229] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:08:39.477411 13807 raft_consensus.cc:2258] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:08:39.479494 13805 ts_tablet_manager.cc:1905] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:08:39.480314 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 13314
I20260620 03:08:39.481065 13807 ts_tablet_manager.cc:1905] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:08:39.489094 13807 ts_tablet_manager.cc:1918] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:08:39.490984 13806 tablet_replica.cc:330] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff: stopping tablet replica
I20260620 03:08:39.491520 13806 raft_consensus.cc:2229] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:08:39.492085 13806 raft_consensus.cc:2258] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:08:39.495551 13806 ts_tablet_manager.cc:1905] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:08:39.496820 13805 ts_tablet_manager.cc:1918] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:08:39.497052 13807 log.cc:1196] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/5fdab43e13714ca6aaafff998a6d47ab
I20260620 03:08:39.497707 13807 ts_tablet_manager.cc:1939] T 5fdab43e13714ca6aaafff998a6d47ab P 303a3ced4f8541ae9bcc3b25611bec62: Deleting consensus metadata
I20260620 03:08:39.499743 13263 catalog_manager.cc:4841] TS 303a3ced4f8541ae9bcc3b25611bec62 (127.0.97.3:45325): tablet 5fdab43e13714ca6aaafff998a6d47ab (table test-table [id=1c6d5a663f3d40d197ad99bc05fb66bf]) successfully deleted
I20260620 03:08:39.500200 13805 log.cc:1196] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/5fdab43e13714ca6aaafff998a6d47ab
I20260620 03:08:39.500800 13805 ts_tablet_manager.cc:1939] T 5fdab43e13714ca6aaafff998a6d47ab P 2c831e7bb0c440299b2482eb8695facf: Deleting consensus metadata
I20260620 03:08:39.502566 13261 catalog_manager.cc:4841] TS 2c831e7bb0c440299b2482eb8695facf (127.0.97.2:35187): tablet 5fdab43e13714ca6aaafff998a6d47ab (table test-table [id=1c6d5a663f3d40d197ad99bc05fb66bf]) successfully deleted
I20260620 03:08:39.503129 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.481980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.0.97.1:37675)
0620 03:08:39.482153 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:39.482161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:39.482180 (+ 19us) server_negotiation.cc:407] Connection header received
0620 03:08:39.482367 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:39.482372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:39.482475 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:39.482587 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:39.489371 (+ 6784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.490617 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.492424 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.493062 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.493188 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:39.496545 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:39.496559 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:39.496577 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:39.496623 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:39.500146 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:39.500946 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:39.500956 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:39.500960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:39.501114 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:39.501492 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:39.501497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:39.501501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:39.501944 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:39.502243 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:39.502521 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 03:08:39.502736 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:08:39.504001 13469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:39.657421 13806 ts_tablet_manager.cc:1918] T 5fdab43e13714ca6aaafff998a6d47ab P e360b58de2934d0e90cf908159b0b5ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:08:39.658267 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 13314
W20260620 03:08:39.679951 13262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:37675 (error 108)
I20260620 03:08:39.681310 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 13478
I20260620 03:08:39.695873 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.681250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37675 (local address 127.0.0.1:42826)
0620 03:08:39.695529 (+ 14279us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:39.695679 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14124,"threads_started":1}
W20260620 03:08:39.696410 13262 catalog_manager.cc:4584] TS e360b58de2934d0e90cf908159b0b5ff (127.0.97.1:37675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5fdab43e13714ca6aaafff998a6d47ab: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37675: connect: Connection refused (error 111)
I20260620 03:08:39.705864 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.682751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.0.97.2:35187)
0620 03:08:39.682930 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:39.682936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:39.682955 (+ 19us) server_negotiation.cc:407] Connection header received
0620 03:08:39.683028 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:39.683033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:39.683120 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:39.683218 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:39.685148 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.686395 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.688373 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.689022 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.689148 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:39.698204 (+ 9056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:39.698222 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:39.698244 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:39.698303 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:39.702510 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:39.703571 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:39.703581 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:39.703584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:39.703780 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:39.704196 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:39.704201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:39.704203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:39.704655 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:39.704961 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:39.705254 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 03:08:39.705455 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:08:39.706635 13614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:39.750275 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.749764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37675 (local address 127.0.0.1:42832)
0620 03:08:39.749990 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:39.750111 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 03:08:39.751433 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.750983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37675 (local address 127.0.0.1:42844)
0620 03:08:39.751141 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:39.751240 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 03:08:39.812130 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.811612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37675 (local address 127.0.0.1:42854)
0620 03:08:39.811853 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:39.811957 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 03:08:39.813284 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.812865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37675 (local address 127.0.0.1:42860)
0620 03:08:39.813025 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:39.813113 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 03:08:39.832471 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 13478
I20260620 03:08:39.847321 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 13623
I20260620 03:08:39.861719 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.848095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60026 (local address 127.0.97.3:45325)
0620 03:08:39.848258 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:39.848263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:39.848283 (+ 20us) server_negotiation.cc:407] Connection header received
0620 03:08:39.848480 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:39.848484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:39.848557 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:39.848648 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:39.850245 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.850931 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.852788 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:39.853145 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:39.853232 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:39.856300 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:39.856314 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:39.856330 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:39.856360 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:39.859084 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:39.859847 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:39.859853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:39.859855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:39.859982 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:39.860367 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:39.860370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:39.860372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:39.860693 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:39.860911 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:39.861240 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 03:08:39.861395 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:08:39.862421 13748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:39.894407 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.893955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37675 (local address 127.0.0.1:42876)
0620 03:08:39.894156 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:39.894250 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 03:08:39.895581 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:39.895090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37675 (local address 127.0.0.1:42886)
0620 03:08:39.895287 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:39.895451 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 03:08:39.951473 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 13623
I20260620 03:08:39.968335 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 13244
I20260620 03:08:39.969516 13307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:40.079708 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:40.078902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37675 (local address 127.0.0.1:42900)
0620 03:08:40.079386 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:40.079496 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303}
I20260620 03:08:40.079710 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 13244
I20260620 03:08:40.101043 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 03:08:40.436348 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 03:08:40.436597 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 03:08:40.782130 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 03:08:40.782 UTC [12296] LOG: received smart shutdown request
2026-06-20 03:08:40.786 UTC [12296] LOG: background worker "logical replication launcher" (PID 12305) exited with exit code 1
2026-06-20 03:08:40.786 UTC [12300] LOG: shutting down
2026-06-20 03:08:40.983 UTC [12296] LOG: database system is shut down
2026-06-20T03:08:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51654 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:08:41 dist-test-slave-xv68 krb5kdc[13822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:08:41 dist-test-slave-xv68 krb5kdc[13822](info): set up 2 sockets
Jun 20 03:08:41 dist-test-slave-xv68 krb5kdc[13822](info): commencing operation
krb5kdc: starting...
W20260620 03:08:42.219767 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:08:42 dist-test-slave-xv68 krb5kdc[13822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:08:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:08:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:08:42.450856 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46875
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46875,127.0.97.61:45489,127.0.97.60:44697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:08:42.775887 13835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:42.776194 13835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:42.776288 13835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:42.783489 13835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:08:42.783596 13835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:42.783643 13835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:42.783702 13835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:08:42.783747 13835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:08:42.793654 13835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46875,127.0.97.61:45489,127.0.97.60:44697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:42.794945 13835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:42.796216 13835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:42.808308 13841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:42.808269 13843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:42.808351 13840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:42.813014 13835 server_base.cc:1034] running on GCE node
I20260620 03:08:42.813784 13835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:42.815418 13835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:42.816641 13835 hybrid_clock.cc:648] HybridClock initialized: now 1781924922816596 us; error 36 us; skew 500 ppm
I20260620 03:08:42.817044 13835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:42 dist-test-slave-xv68 krb5kdc[13822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:42.825738 13835 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:08:42.827764 13835 webserver.cc:460] Webserver started at http://127.0.97.62:32813/ using document root <none> and password file <none>
I20260620 03:08:42.828526 13835 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:42.828677 13835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:42.828990 13835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:08:42.832073 13835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "a97c1c6423d84385bd9f4f41adc33b6c"
format_stamp: "Formatted at 2026-06-20 03:08:42 on dist-test-slave-xv68"
server_key: "a1158e6d1621e4e8e814ee17977a0151"
server_key_iv: "79f254fb4370315d5a5465a55edd1bbf"
server_key_version: "encryptionkey@0"
I20260620 03:08:42.832859 13835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "a97c1c6423d84385bd9f4f41adc33b6c"
format_stamp: "Formatted at 2026-06-20 03:08:42 on dist-test-slave-xv68"
server_key: "a1158e6d1621e4e8e814ee17977a0151"
server_key_iv: "79f254fb4370315d5a5465a55edd1bbf"
server_key_version: "encryptionkey@0"
I20260620 03:08:42.838131 13835 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:08:42.841658 13850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:42.842921 13835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 03:08:42.843125 13835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "a97c1c6423d84385bd9f4f41adc33b6c"
format_stamp: "Formatted at 2026-06-20 03:08:42 on dist-test-slave-xv68"
server_key: "a1158e6d1621e4e8e814ee17977a0151"
server_key_iv: "79f254fb4370315d5a5465a55edd1bbf"
server_key_version: "encryptionkey@0"
I20260620 03:08:42.843349 13835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:42.863193 13835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:42.867434 13835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:42.867826 13835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:42.887248 13835 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46875
I20260620 03:08:42.887267 13901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46875 every 8 connection(s)
I20260620 03:08:42.888868 13835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:08:42.892725 13902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:08:42.892859 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 13835
I20260620 03:08:42.893066 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:08:42.893851 388 external_mini_cluster.cc:1428] Setting key 8b3fa4473c0bcec2c23ec43dbd502b7b
I20260620 03:08:42.902828 13902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Jun 20 03:08:42 dist-test-slave-xv68 krb5kdc[13822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:08:42.927158 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:42.907692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45904 (local address 127.0.97.62:46875)
0620 03:08:42.908461 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:42.908469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:42.908525 (+ 56us) server_negotiation.cc:407] Connection header received
0620 03:08:42.909653 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:42.909659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:42.909886 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:42.910097 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:42.911347 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:42.912675 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:42.914079 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:42.914605 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:42.914810 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:42.919711 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:42.919720 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:42.919733 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:42.919763 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:42.922622 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:42.923574 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:42.923580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:42.923583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:42.923746 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:42.924265 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:42.924273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:42.924279 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:42.924972 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:42.925137 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:42.925824 (+ 687us) server_negotiation.cc:299] Negotiation successful
0620 03:08:42.926106 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":392,"threads_started":1}
I20260620 03:08:42.927191 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:42.906307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46875 (local address 127.0.0.1:45904)
0620 03:08:42.907120 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:42.907163 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:42.909270 (+ 2107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:42.910103 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:42.910109 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:42.910799 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:42.911127 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:42.911144 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:42.912831 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:42.912838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:42.913911 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:42.913922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:42.914825 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:42.914832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:42.915016 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:42.915756 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:42.915777 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:42.919431 (+ 3654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:42.922771 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:42.922786 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:42.922803 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:42.923444 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:42.923866 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:42.923873 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:42.923880 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:42.924138 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:42.925140 (+ 1002us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:42.925150 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:42.925420 (+ 270us) client_negotiation.cc:769] Sending connection context
0620 03:08:42.925715 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 03:08:42.926328 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":67,"thread_start_us":374,"threads_started":1}
I20260620 03:08:42.930751 13902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:08:42.932004 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:42.931515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45489 (local address 127.0.0.1:36126)
0620 03:08:42.931713 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:42.931850 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 03:08:42.932281 13852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45489: connect: Connection refused (error 111)
I20260620 03:08:42.933985 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:42.933522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45489 (local address 127.0.0.1:36138)
0620 03:08:42.933705 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:42.933796 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 03:08:42.934199 13902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45489: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45489: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 03:08:42.946961 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45489
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46875,127.0.97.61:45489,127.0.97.60:44697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:08:42.983572 13902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } attempt: 1
I20260620 03:08:42.984869 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:42.984397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45489 (local address 127.0.0.1:36154)
0620 03:08:42.984626 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:42.984718 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 03:08:42.986167 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:42.985770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45489 (local address 127.0.0.1:36166)
0620 03:08:42.985934 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:42.986006 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 03:08:42.986488 13902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45489: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45489: connect: Connection refused (error 111)
I20260620 03:08:43.054766 13902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } attempt: 2
I20260620 03:08:43.056130 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.055617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45489 (local address 127.0.0.1:36172)
0620 03:08:43.055858 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.055973 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 03:08:43.057377 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.056965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45489 (local address 127.0.0.1:36178)
0620 03:08:43.057130 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.057237 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 03:08:43.057598 13902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45489: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45489: connect: Connection refused (error 111)
I20260620 03:08:43.148881 13902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } attempt: 3
I20260620 03:08:43.150089 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.149624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45489 (local address 127.0.0.1:36192)
0620 03:08:43.149827 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.149926 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 03:08:43.151436 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.150948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45489 (local address 127.0.0.1:36200)
0620 03:08:43.151119 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.151212 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 03:08:43.151731 13902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45489: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45489: connect: Connection refused (error 111)
W20260620 03:08:43.283340 13909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:43.283645 13909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:43.283811 13909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:43.291253 13909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:08:43.291390 13909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:43.291529 13909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:43.291762 13909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:08:43.291872 13909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:08:43.295022 13902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } attempt: 4
I20260620 03:08:43.296300 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.295762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45489 (local address 127.0.0.1:36216)
0620 03:08:43.296077 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.296171 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260620 03:08:43.297345 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.296969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45489 (local address 127.0.0.1:36226)
0620 03:08:43.297131 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.297230 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 03:08:43.297562 13902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45489: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45489: connect: Connection refused (error 111)
I20260620 03:08:43.302614 13909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46875,127.0.97.61:45489,127.0.97.60:44697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:43.304009 13909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:43.305249 13909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:43.318293 13914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:43.318238 13917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:43.318256 13915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:43.319231 13909 server_base.cc:1034] running on GCE node
I20260620 03:08:43.320202 13909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:43.321928 13909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:43.323220 13909 hybrid_clock.cc:648] HybridClock initialized: now 1781924923323222 us; error 106 us; skew 500 ppm
I20260620 03:08:43.323644 13909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:43 dist-test-slave-xv68 krb5kdc[13822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:43.327056 13909 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 03:08:43.328840 13909 webserver.cc:460] Webserver started at http://127.0.97.61:39693/ using document root <none> and password file <none>
I20260620 03:08:43.329571 13909 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:43.329720 13909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:43.330039 13909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:08:43.333290 13909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-1/data/instance:
uuid: "1a72567fd96847b89eb6e1c4aacaafc1"
format_stamp: "Formatted at 2026-06-20 03:08:43 on dist-test-slave-xv68"
server_key: "28b3bff33f1945e247f12077ae5a4f5b"
server_key_iv: "267c01310a464de280a3272d256fb1d7"
server_key_version: "encryptionkey@0"
I20260620 03:08:43.334622 13909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-1/wal/instance:
uuid: "1a72567fd96847b89eb6e1c4aacaafc1"
format_stamp: "Formatted at 2026-06-20 03:08:43 on dist-test-slave-xv68"
server_key: "28b3bff33f1945e247f12077ae5a4f5b"
server_key_iv: "267c01310a464de280a3272d256fb1d7"
server_key_version: "encryptionkey@0"
I20260620 03:08:43.340715 13909 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260620 03:08:43.344261 13924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:43.345800 13909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:08:43.345999 13909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-1/wal
uuid: "1a72567fd96847b89eb6e1c4aacaafc1"
format_stamp: "Formatted at 2026-06-20 03:08:43 on dist-test-slave-xv68"
server_key: "28b3bff33f1945e247f12077ae5a4f5b"
server_key_iv: "267c01310a464de280a3272d256fb1d7"
server_key_version: "encryptionkey@0"
I20260620 03:08:43.346223 13909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:43.365298 13909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:43.370191 13909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:43.370499 13909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:43.393520 13909 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:45489
I20260620 03:08:43.393613 13975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:45489 every 8 connection(s)
I20260620 03:08:43.396270 13909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-1/data/info.pb
I20260620 03:08:43.400043 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 13909
I20260620 03:08:43.400224 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-1/wal/instance
I20260620 03:08:43.400885 388 external_mini_cluster.cc:1428] Setting key 029995d915336fc86ddb0a5d84706571
I20260620 03:08:43.401510 13976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:08:43.411774 13976 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 20 03:08:43 dist-test-slave-xv68 krb5kdc[13822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:08:43.436939 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.414859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.0.97.62:46875)
0620 03:08:43.415098 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:43.415105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:43.416770 (+ 1665us) server_negotiation.cc:407] Connection header received
0620 03:08:43.418739 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:43.418745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:43.418845 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:43.418965 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:43.420592 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.421278 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.422837 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.423256 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.423423 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:43.429922 (+ 6499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:43.429934 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:43.429939 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:43.429981 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:43.433287 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:43.434290 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:43.434302 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:43.434305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:43.434443 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:43.434919 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:43.434931 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:43.434934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:43.435416 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:43.435591 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:43.436338 (+ 747us) server_negotiation.cc:299] Negotiation successful
0620 03:08:43.436589 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 03:08:43.437983 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.415664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46875 (local address 127.0.0.1:45906)
0620 03:08:43.416577 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.416623 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:43.418443 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:43.419216 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:43.419225 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:43.419818 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:43.420408 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:43.420426 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.421445 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.421455 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:43.422678 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:43.422695 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.423556 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.423569 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:43.423961 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:43.424835 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:43.424879 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:43.429166 (+ 4287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:43.433504 (+ 4338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:43.433518 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:43.433533 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:43.434024 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:43.434568 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:43.434574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:43.434577 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:43.434782 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:43.435554 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:43.435564 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:43.435956 (+ 392us) client_negotiation.cc:769] Sending connection context
0620 03:08:43.436239 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 03:08:43.436793 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":418,"threads_started":1}
I20260620 03:08:43.440151 13976 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:08:43 dist-test-slave-xv68 krb5kdc[13822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:08:43.457348 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.440951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45489 (local address 127.0.0.1:36236)
0620 03:08:43.441207 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.441222 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:43.441305 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:43.442402 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:43.442406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:43.442895 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:43.443144 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:43.443154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.444579 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.444585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:43.445408 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:43.445418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.446363 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.446368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:43.446635 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:43.447331 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:43.447360 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:43.449844 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:43.454050 (+ 4206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:43.454055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:43.454059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:43.454478 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:43.455114 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:43.455118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:43.455120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:43.455280 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:43.456157 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:43.456161 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:43.456321 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 03:08:43.456521 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 03:08:43.456989 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":24}
I20260620 03:08:43.457870 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.441245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36236 (local address 127.0.97.61:45489)
0620 03:08:43.441993 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:43.441999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:43.442024 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:08:43.442067 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:43.442073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:43.442252 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:43.442463 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:43.443340 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.444420 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.445539 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.446180 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.446423 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:43.450161 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:43.450173 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:43.450187 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:43.450217 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:43.453861 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:43.454692 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:43.454699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:43.454701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:43.454901 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:43.455557 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:43.455561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:43.455563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:43.456007 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:43.456190 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:43.456763 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 03:08:43.457385 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":588,"thread_start_us":462,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 03:08:43.460729 13976 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:08:43.460965 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44697
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46875,127.0.97.61:45489,127.0.97.60:44697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:08:43.462764 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.461996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34724)
0620 03:08:43.462329 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.462533 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260620 03:08:43.463382 13928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
I20260620 03:08:43.465303 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.464876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34740)
0620 03:08:43.465079 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.465171 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260620 03:08:43.465736 13976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44697: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
I20260620 03:08:43.515050 13976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } attempt: 1
I20260620 03:08:43.516479 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.516009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34750)
0620 03:08:43.516230 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.516329 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 03:08:43.517666 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.517287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34758)
0620 03:08:43.517482 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.517551 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 03:08:43.517859 13976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44697: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
I20260620 03:08:43.586215 13976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } attempt: 2
I20260620 03:08:43.587406 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.586871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34766)
0620 03:08:43.587102 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.587200 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 03:08:43.588353 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.588025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34772)
0620 03:08:43.588171 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.588244 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 03:08:43.588546 13976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44697: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
I20260620 03:08:43.596863 13902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } attempt: 5
Jun 20 03:08:43 dist-test-slave-xv68 krb5kdc[13822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:08:43.614166 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.597699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36242 (local address 127.0.97.61:45489)
0620 03:08:43.597883 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:43.597887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:43.597961 (+ 74us) server_negotiation.cc:407] Connection header received
0620 03:08:43.598211 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:43.598216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:43.598291 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:43.598381 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:43.599466 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.600404 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.602275 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.602902 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.603018 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:43.607645 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:43.607650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:43.607654 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:43.607690 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:43.610864 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:43.611899 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:43.611905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:43.611908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:43.612042 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:43.612688 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:43.612692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:43.612694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:43.613027 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:43.613193 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:43.613607 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 03:08:43.613826 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 03:08:43.614166 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.597626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45489 (local address 127.0.0.1:36242)
0620 03:08:43.597849 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.597872 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:43.598023 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:43.598459 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:43.598464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:43.598976 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:43.599259 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:43.599272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.600572 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.600578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:43.601993 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:43.602026 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.603121 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.603128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:43.603351 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:43.604169 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:43.604193 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:43.607403 (+ 3210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:43.611247 (+ 3844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:43.611254 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:43.611260 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:43.611756 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:43.612334 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:43.612341 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:43.612345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:43.612546 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:43.613156 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:43.613161 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:43.613324 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 03:08:43.613484 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 03:08:43.613797 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 03:08:43.615447 13902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:08:43.616736 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.616310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34782)
0620 03:08:43.616503 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.616591 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 03:08:43.618355 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.617862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34790)
0620 03:08:43.618144 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.618237 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 03:08:43.618822 13902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44697: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
I20260620 03:08:43.670262 13902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } attempt: 1
I20260620 03:08:43.672086 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.671250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34792)
0620 03:08:43.671679 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.671821 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260620 03:08:43.673593 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.673100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34800)
0620 03:08:43.673309 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.673418 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 03:08:43.673883 13902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44697: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
I20260620 03:08:43.679780 13976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } attempt: 3
I20260620 03:08:43.680972 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.680450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34810)
0620 03:08:43.680668 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.680770 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 03:08:43.682219 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.681730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34812)
0620 03:08:43.681983 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.682079 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260620 03:08:43.682463 13976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44697: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
I20260620 03:08:43.742177 13902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } attempt: 2
I20260620 03:08:43.743491 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.742872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34824)
0620 03:08:43.743139 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.743256 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 03:08:43.744621 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.744224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34834)
0620 03:08:43.744398 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.744488 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 03:08:43.744910 13902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44697: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
W20260620 03:08:43.818197 13983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:43.818567 13983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:43.818739 13983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 03:08:43.825690 13976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } attempt: 4
W20260620 03:08:43.826701 13983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:08:43.826829 13983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:43.826916 13983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260620 03:08:43.826992 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.826428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34844)
0620 03:08:43.826678 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.826787 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 03:08:43.827008 13983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:08:43.827070 13983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:08:43.828027 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.827685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34852)
0620 03:08:43.827831 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.827912 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 03:08:43.828286 13976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44697: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
I20260620 03:08:43.837695 13983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46875,127.0.97.61:45489,127.0.97.60:44697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:43.839035 13983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:43.840436 13983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 03:08:43.851137 13902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } attempt: 3
I20260620 03:08:43.852566 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.851827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34860)
0620 03:08:43.852236 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.852343 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
W20260620 03:08:43.852659 13988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:43.853194 13991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:43.853724 13983 server_base.cc:1034] running on GCE node
I20260620 03:08:43.854296 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.853668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34874)
0620 03:08:43.854019 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.854105 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223}
W20260620 03:08:43.854403 13989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:43.854560 13902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44697: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44697: connect: Connection refused (error 111)
I20260620 03:08:43.855480 13983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:43.856920 13983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:43.858186 13983 hybrid_clock.cc:648] HybridClock initialized: now 1781924923858113 us; error 65 us; skew 500 ppm
I20260620 03:08:43.858572 13983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:43 dist-test-slave-xv68 krb5kdc[13822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:43.861903 13983 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 03:08:43.863711 13983 webserver.cc:460] Webserver started at http://127.0.97.60:45403/ using document root <none> and password file <none>
I20260620 03:08:43.864450 13983 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:43.864604 13983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:43.864936 13983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:08:43.868180 13983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-2/data/instance:
uuid: "47fa4e000b12460c909ed915cf74228f"
format_stamp: "Formatted at 2026-06-20 03:08:43 on dist-test-slave-xv68"
server_key: "92a28cdb1b9d1454b1ddf494d377b60c"
server_key_iv: "b7cd0f0d4a4025e806c1a79588caf66b"
server_key_version: "encryptionkey@0"
I20260620 03:08:43.869006 13983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-2/wal/instance:
uuid: "47fa4e000b12460c909ed915cf74228f"
format_stamp: "Formatted at 2026-06-20 03:08:43 on dist-test-slave-xv68"
server_key: "92a28cdb1b9d1454b1ddf494d377b60c"
server_key_iv: "b7cd0f0d4a4025e806c1a79588caf66b"
server_key_version: "encryptionkey@0"
I20260620 03:08:43.874377 13983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:08:43.878008 13998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:43.879567 13983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:08:43.879752 13983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-2/wal
uuid: "47fa4e000b12460c909ed915cf74228f"
format_stamp: "Formatted at 2026-06-20 03:08:43 on dist-test-slave-xv68"
server_key: "92a28cdb1b9d1454b1ddf494d377b60c"
server_key_iv: "b7cd0f0d4a4025e806c1a79588caf66b"
server_key_version: "encryptionkey@0"
I20260620 03:08:43.879966 13983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:43.901670 13983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:43.906185 13983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:43.906486 13983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:43.925817 13983 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44697
I20260620 03:08:43.925858 14049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44697 every 8 connection(s)
I20260620 03:08:43.927397 13983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-2/data/info.pb
I20260620 03:08:43.931111 14050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:08:43.936910 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 13983
I20260620 03:08:43.937108 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924720365844-388-0/minicluster-data/master-2/wal/instance
I20260620 03:08:43.937646 388 external_mini_cluster.cc:1428] Setting key b888a6f131b73e7e9bf7debef95d9c26
I20260620 03:08:43.937969 14050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:08:43 dist-test-slave-xv68 krb5kdc[13822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 03:08:43 dist-test-slave-xv68 krb5kdc[13822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 03:08:43.959225 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.940893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45908 (local address 127.0.97.62:46875)
0620 03:08:43.941462 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:43.941469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:43.942057 (+ 588us) server_negotiation.cc:407] Connection header received
0620 03:08:43.942969 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:43.942978 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:43.943064 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:43.943184 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:43.944962 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.946216 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.948100 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.948734 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.948831 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:43.952917 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:43.952923 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:43.952926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:43.952961 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:43.956078 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:43.956906 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:43.956912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:43.956915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:43.957044 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:43.957460 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:43.957465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:43.957474 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:43.957738 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:43.957891 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:43.958597 (+ 706us) server_negotiation.cc:299] Negotiation successful
0620 03:08:43.958820 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":302,"threads_started":1}
I20260620 03:08:43.960431 14055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.940880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46875 (local address 127.0.0.1:45908)
0620 03:08:43.941900 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.941946 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:43.942716 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:43.943270 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:43.943277 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:43.944183 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:43.944750 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:43.944782 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.946414 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.946419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:43.947953 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:43.947965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.948988 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.948996 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:43.949207 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:43.950043 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:43.950065 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:43.952697 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:43.956226 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:43.956238 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:43.956252 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:43.956773 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:43.957161 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:43.957166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:43.957170 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:43.957341 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:43.957866 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:43.957879 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:43.958221 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 03:08:43.958482 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 03:08:43.959008 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"thread_start_us":599,"threads_started":1}
I20260620 03:08:43.960455 14053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.939755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34890 (local address 127.0.97.60:44697)
0620 03:08:43.940559 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:43.940568 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:43.940605 (+ 37us) server_negotiation.cc:407] Connection header received
0620 03:08:43.942701 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:43.942723 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:43.943013 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:43.943229 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:43.945074 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.946870 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.948593 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.949299 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.949457 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:43.954098 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:43.954108 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:43.954120 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:43.954153 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:43.957135 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:43.957828 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:43.957834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:43.957836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:43.957959 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:43.958329 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:43.958333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:43.958335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:43.958683 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:43.958994 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:43.959471 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 03:08:43.959714 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":384,"threads_started":1}
I20260620 03:08:43.962086 14050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:08:43 dist-test-slave-xv68 krb5kdc[13822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 03:08:43 dist-test-slave-xv68 krb5kdc[13822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:08:43.975900 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.962859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36254 (local address 127.0.97.61:45489)
0620 03:08:43.963036 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:43.963041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:43.963114 (+ 73us) server_negotiation.cc:407] Connection header received
0620 03:08:43.963340 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:43.963346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:43.963416 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:43.963524 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:43.964328 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.965019 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.966191 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.966609 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.966737 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:43.969990 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:43.969995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:43.969998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:43.970031 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:43.972871 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:43.973771 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:43.973775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:43.973778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:43.973919 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:43.974388 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:43.974392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:43.974394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:43.974701 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:43.974813 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:43.975373 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 03:08:43.975593 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 03:08:43.975983 14055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.962698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45489 (local address 127.0.0.1:36254)
0620 03:08:43.963046 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.963065 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:43.963173 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:43.963545 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:43.963549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:43.963996 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:43.964194 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:43.964203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.965164 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.965169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:43.966053 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:43.966064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.966756 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.966762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:43.966987 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:43.967612 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:43.967629 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:43.969784 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:43.973057 (+ 3273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:43.973064 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:43.973070 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:43.973626 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:43.974070 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:43.974073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:43.974075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:43.974274 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:43.974827 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:43.974832 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:43.975044 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 03:08:43.975225 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 03:08:43.975585 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247}
I20260620 03:08:43.977111 14050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:08:43.978703 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.963681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45916 (local address 127.0.97.62:46875)
0620 03:08:43.963837 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:43.963841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:43.963906 (+ 65us) server_negotiation.cc:407] Connection header received
0620 03:08:43.964125 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:43.964128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:43.964212 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:43.964339 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:43.965593 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.966259 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.967708 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.968127 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.968264 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:43.972327 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:43.972332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:43.972336 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:43.972369 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:43.975737 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:43.976586 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:43.976597 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:43.976599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:43.976714 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:43.977189 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:43.977194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:43.977198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:43.977556 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:43.977838 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:43.978109 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 03:08:43.978361 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 03:08:43 dist-test-slave-xv68 krb5kdc[13822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 03:08:43 dist-test-slave-xv68 krb5kdc[13822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:08:43.992655 14053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.977850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34904 (local address 127.0.97.60:44697)
0620 03:08:43.978037 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:43.978043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:43.978228 (+ 185us) server_negotiation.cc:407] Connection header received
0620 03:08:43.978373 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:43.978378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:43.978448 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:43.978540 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:43.979563 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.980459 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.981608 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.982273 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.982412 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:43.986142 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:43.986149 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:43.986154 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:43.986190 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:43.989432 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:43.990274 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:43.990278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:43.990281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:43.990479 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:43.991022 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:43.991029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:43.991033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:43.991458 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:43.991626 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:43.992007 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 03:08:43.992241 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 03:08:43.992667 14055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.977850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34904)
0620 03:08:43.978138 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:43.978154 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:43.978259 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:43.978655 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:43.978660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:43.979163 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:43.979380 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:43.979397 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.980619 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.980622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:43.981468 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:43.981477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.982456 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.982461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:43.982716 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:43.983390 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:43.983407 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:43.985938 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:43.989597 (+ 3659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:43.989601 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:43.989603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:43.990142 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:43.990750 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:43.990757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:43.990761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:43.990912 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:43.991597 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:43.991601 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:43.991753 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 03:08:43.991876 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 03:08:43.992389 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":147}
I20260620 03:08:43.997291 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:43.980495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36266 (local address 127.0.97.61:45489)
0620 03:08:43.980673 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:43.980679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:43.980696 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:08:43.980817 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:43.980822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:43.980890 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:43.980988 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:43.982433 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.983517 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.985252 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:43.985802 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:43.985903 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:43.991319 (+ 5416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:43.991326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:43.991330 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:43.991374 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:43.994544 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:43.995474 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:43.995478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:43.995480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:43.995608 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:43.995995 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:43.995998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:43.996000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:43.996266 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:43.996465 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:43.996827 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 03:08:43.996993 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 03:08:43.998850 14050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f: Bootstrap starting.
I20260620 03:08:44.002218 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:08:44.002748 14050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f: Neither blocks nor log segments found. Creating new log.
I20260620 03:08:44.003834 14050 log.cc:824] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:44.010098 14050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f: No bootstrap required, opened a new log
I20260620 03:08:44.022945 14050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } }
I20260620 03:08:44.023456 14050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:08:44.023589 14050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47fa4e000b12460c909ed915cf74228f, State: Initialized, Role: FOLLOWER
I20260620 03:08:44.024495 14050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } }
I20260620 03:08:44.027796 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:44.007435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.0.97.62:46875)
0620 03:08:44.007628 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:44.007634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:44.008360 (+ 726us) server_negotiation.cc:407] Connection header received
0620 03:08:44.008680 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:44.008686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:44.008771 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:44.008845 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:44.011458 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:44.012555 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:44.014785 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:44.015465 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:44.015567 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:44.019796 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:44.019803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:44.019809 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:44.019865 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:44.023908 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:44.024722 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:44.024728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:44.024731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:44.024873 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:44.025417 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:44.025425 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:44.025431 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:44.025893 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:44.026200 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:44.026930 (+ 730us) server_negotiation.cc:299] Negotiation successful
0620 03:08:44.027254 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 03:08:44.030269 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:44.008808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.0.97.61:45489)
0620 03:08:44.009037 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:44.009045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:44.011347 (+ 2302us) server_negotiation.cc:407] Connection header received
0620 03:08:44.011585 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:44.011590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:44.011664 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:44.011732 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:44.014475 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:44.015621 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:44.018143 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:44.018817 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:44.018916 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:44.022589 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:44.022596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:44.022600 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:44.022641 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:44.026477 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:44.027696 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:44.027702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:44.027705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:44.027861 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:44.028557 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:44.028562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:44.028565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:44.028926 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:44.029151 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:44.029686 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 03:08:44.029923 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 03:08:44.030525 14053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:44.007397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34910 (local address 127.0.97.60:44697)
0620 03:08:44.007638 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:44.007644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:44.011032 (+ 3388us) server_negotiation.cc:407] Connection header received
0620 03:08:44.011591 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:44.011596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:44.011665 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:44.011732 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:44.014476 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:44.015663 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:44.018143 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:44.018948 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:44.019042 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:44.022626 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:44.022632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:44.022635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:44.022670 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:44.026663 (+ 3993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:44.027697 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:44.027702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:44.027706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:44.027863 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:44.028283 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:44.028288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:44.028293 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:44.028704 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:44.028945 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:44.029940 (+ 995us) server_negotiation.cc:299] Negotiation successful
0620 03:08:44.030143 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 03:08:44.031754 13902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } attempt: 4
I20260620 03:08:44.032809 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } } }
I20260620 03:08:44.033241 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:08:44.033708 14050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 03:08:44 dist-test-slave-xv68 krb5kdc[13822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260620 03:08:44.045331 14075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 03:08:44.045459 14075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 03:08:44.049803 14053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:44.032317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34914 (local address 127.0.97.60:44697)
0620 03:08:44.032469 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:44.032475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:44.032601 (+ 126us) server_negotiation.cc:407] Connection header received
0620 03:08:44.032729 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:44.032734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:44.032802 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:44.032906 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:44.033683 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:44.034553 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:44.036012 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:44.036637 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:44.036791 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:44.040983 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:44.040989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:44.040994 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:44.041030 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:44.044977 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:44.045804 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:44.045810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:44.045813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:44.045946 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:44.048236 (+ 2290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:44.048241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:44.048244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:44.048638 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:44.048827 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:44.049224 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 03:08:44.049442 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 03:08:44.049803 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:44.032280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34914)
0620 03:08:44.032506 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:44.032523 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:44.032633 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:44.032924 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:44.032929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:44.033401 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:44.033550 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:44.033558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:44.034687 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:44.034693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:44.035869 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:44.035881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:44.036805 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:44.036811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:44.037000 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:44.037982 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:44.038004 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:44.040782 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:44.045138 (+ 4356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:44.045145 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:44.045149 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:44.045677 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:44.047898 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:44.047903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:44.047907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:44.048079 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:44.048785 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:44.048790 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:44.048962 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 03:08:44.049108 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 03:08:44.049413 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260620 03:08:44.055284 13902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c: Bootstrap starting.
I20260620 03:08:44.042047 13983 master_runner.cc:186] Error getting master registration for 127.0.97.62:46875: OK, instance_id { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" instance_seqno: 1781924922881728 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 03:08:44.055881 13983 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 03:08:44.058698 13902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c: Neither blocks nor log segments found. Creating new log.
I20260620 03:08:44.060042 13902 log.cc:824] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:44.063033 13902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c: No bootstrap required, opened a new log
I20260620 03:08:44.071543 13902 raft_consensus.cc:348] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } }
I20260620 03:08:44.071887 13902 raft_consensus.cc:374] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:08:44.072057 13902 raft_consensus.cc:729] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97c1c6423d84385bd9f4f41adc33b6c, State: Initialized, Role: FOLLOWER
I20260620 03:08:44.072715 13902 consensus_queue.cc:260] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } }
I20260620 03:08:44.073733 14078 sys_catalog.cc:455] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } } }
I20260620 03:08:44.074136 14078 sys_catalog.cc:458] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:08:44.074950 13902 sys_catalog.cc:564] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [sys.catalog]: configured and running, proceeding with master startup.
W20260620 03:08:44.094125 14089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 03:08:44.094435 14089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 03:08:44.086395 13835 master_runner.cc:186] Error getting master registration for 127.0.97.61:45489: OK, instance_id { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" instance_seqno: 1781924923387665 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 03:08:44.102133 13835 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 03:08:44.120932 13856 tablet.cc:2369] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:08:44.127558 13976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } attempt: 5
Jun 20 03:08:44 dist-test-slave-xv68 krb5kdc[13822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 03:08:44.144822 14078 raft_consensus.cc:482] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:08:44.145150 14078 raft_consensus.cc:504] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } }
I20260620 03:08:44.147069 14078 leader_election.cc:288] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a72567fd96847b89eb6e1c4aacaafc1 (127.0.97.61:45489), 47fa4e000b12460c909ed915cf74228f (127.0.97.60:44697)
I20260620 03:08:44.148805 14025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47fa4e000b12460c909ed915cf74228f" is_pre_election: true
I20260620 03:08:44.149555 14025 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97c1c6423d84385bd9f4f41adc33b6c in term 0.
I20260620 03:08:44.150312 13854 leader_election.cc:302] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47fa4e000b12460c909ed915cf74228f, a97c1c6423d84385bd9f4f41adc33b6c; no voters:
I20260620 03:08:44.150753 14078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:08:44.150935 14078 raft_consensus.cc:482] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:08:44.151070 14078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:08:44.151753 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:44.128280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44697 (local address 127.0.0.1:34916)
0620 03:08:44.128703 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:44.128724 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:44.128853 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:44.129867 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:44.129872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:44.130409 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:44.130614 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:44.130626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:44.132193 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:44.132200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:44.133848 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:44.133865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:44.134930 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:44.134937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:44.135157 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:44.135945 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:44.135972 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:44.139080 (+ 3108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:44.143753 (+ 4673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:44.143760 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:44.143764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:44.144347 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:44.148355 (+ 4008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:44.148361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:44.148365 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:44.148544 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:44.149625 (+ 1081us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:44.149630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:44.150751 (+ 1121us) client_negotiation.cc:769] Sending connection context
0620 03:08:44.150967 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 03:08:44.151359 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276}
I20260620 03:08:44.152761 14053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:44.129217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34916 (local address 127.0.97.60:44697)
0620 03:08:44.129418 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:44.129425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:44.129444 (+ 19us) server_negotiation.cc:407] Connection header received
0620 03:08:44.129520 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:44.129528 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:44.129611 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:44.129718 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:44.130798 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:44.132020 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:44.134016 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:44.134761 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:44.134909 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:44.139362 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:44.139371 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:44.139377 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:44.139421 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:44.143526 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:44.148009 (+ 4483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:44.148014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:44.148018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:44.148206 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:44.148677 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:44.148684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:44.148694 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:44.149401 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:44.149562 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:44.152111 (+ 2549us) server_negotiation.cc:299] Negotiation successful
0620 03:08:44.152385 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 03:08:44.158617 14004 tablet.cc:2369] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:08:44.158449 14078 raft_consensus.cc:504] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } }
I20260620 03:08:44.160830 13976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1: Bootstrap starting.
I20260620 03:08:44.165409 13976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1: Neither blocks nor log segments found. Creating new log.
I20260620 03:08:44.165792 14078 leader_election.cc:288] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [CANDIDATE]: Term 1 election: Requested vote from peers 1a72567fd96847b89eb6e1c4aacaafc1 (127.0.97.61:45489), 47fa4e000b12460c909ed915cf74228f (127.0.97.60:44697)
I20260620 03:08:44.165849 14025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47fa4e000b12460c909ed915cf74228f"
I20260620 03:08:44.166079 14025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:08:44.168370 13976 log.cc:824] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:44.169590 14025 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97c1c6423d84385bd9f4f41adc33b6c in term 1.
I20260620 03:08:44.170099 13854 leader_election.cc:302] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47fa4e000b12460c909ed915cf74228f, a97c1c6423d84385bd9f4f41adc33b6c; no voters:
I20260620 03:08:44.170517 14078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:08:44.178403 14078 raft_consensus.cc:686] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [term 1 LEADER]: Becoming Leader. State: Replica: a97c1c6423d84385bd9f4f41adc33b6c, State: Running, Role: LEADER
I20260620 03:08:44.179116 14078 consensus_queue.cc:237] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } }
I20260620 03:08:44.185509 13976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1: No bootstrap required, opened a new log
I20260620 03:08:44.188282 14092 sys_catalog.cc:455] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a97c1c6423d84385bd9f4f41adc33b6c. Latest consensus state: current_term: 1 leader_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } } }
I20260620 03:08:44.189893 14092 sys_catalog.cc:458] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [sys.catalog]: This master's current role is: LEADER
I20260620 03:08:44.192673 14094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:08:44.195322 14094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:08:44.198673 13976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } }
I20260620 03:08:44.199136 13976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:08:44.199263 13976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a72567fd96847b89eb6e1c4aacaafc1, State: Initialized, Role: FOLLOWER
I20260620 03:08:44.200177 13976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } }
I20260620 03:08:44.209820 14096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } } }
I20260620 03:08:44.210237 14096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:08:44.212180 13976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:08:44.212452 14025 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f [term 1 FOLLOWER]: Refusing update from remote peer a97c1c6423d84385bd9f4f41adc33b6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:08:44.212224 13950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a72567fd96847b89eb6e1c4aacaafc1"
I20260620 03:08:44.212759 13950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:08:44.213418 14092 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:08:44.216418 13950 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97c1c6423d84385bd9f4f41adc33b6c in term 1.
I20260620 03:08:44.212250 13951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" is_pre_election: true
I20260620 03:08:44.217082 13951 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a97c1c6423d84385bd9f4f41adc33b6c in term 1. Re-sending same reply.
I20260620 03:08:44.218179 13950 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1 [term 1 FOLLOWER]: Refusing update from remote peer a97c1c6423d84385bd9f4f41adc33b6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:08:44.219143 14092 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:08:44.256228 14096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a97c1c6423d84385bd9f4f41adc33b6c. Latest consensus state: current_term: 1 leader_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } } }
I20260620 03:08:44.256551 14096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:08:44.257267 14095 mvcc.cc:205] Tried to move back new op lower bound from 7298764489561255936 to 7298764489442775040. Current Snapshot: MvccSnapshot[applied={T|T < 7298764489561255936}]
I20260620 03:08:44.259143 14092 sys_catalog.cc:455] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } } }
I20260620 03:08:44.259447 14092 sys_catalog.cc:458] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [sys.catalog]: This master's current role is: LEADER
I20260620 03:08:44.260653 14094 catalog_manager.cc:1347] Generated new cluster ID: ea50c45d0fc848439ca73bba6867024a
I20260620 03:08:44.260780 14094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:08:44.265468 14099 mvcc.cc:205] Tried to move back new op lower bound from 7298764489561255936 to 7298764489442775040. Current Snapshot: MvccSnapshot[applied={T|T < 7298764489561255936}]
I20260620 03:08:44.269819 14078 sys_catalog.cc:455] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } } }
I20260620 03:08:44.270202 14078 sys_catalog.cc:458] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c [sys.catalog]: This master's current role is: LEADER
I20260620 03:08:44.271423 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a97c1c6423d84385bd9f4f41adc33b6c. Latest consensus state: current_term: 1 leader_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } } }
I20260620 03:08:44.271739 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:08:44.272521 14096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } } }
I20260620 03:08:44.272776 14096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:08:44.278776 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97c1c6423d84385bd9f4f41adc33b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46875 } } peers { permanent_uuid: "1a72567fd96847b89eb6e1c4aacaafc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45489 } } peers { permanent_uuid: "47fa4e000b12460c909ed915cf74228f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44697 } } }
I20260620 03:08:44.279064 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:08:44.285260 14094 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:08:44.287206 14094 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:08:44.312839 14094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a97c1c6423d84385bd9f4f41adc33b6c: Generated new TSK 0
I20260620 03:08:44.314046 14094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:08:44.337306 14116 catalog_manager.cc:1259] Loaded cluster ID: ea50c45d0fc848439ca73bba6867024a
I20260620 03:08:44.337435 14116 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1: loading cluster ID for follower catalog manager: success
I20260620 03:08:44.341265 14116 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1: acquiring CA information for follower catalog manager: success
I20260620 03:08:44.343777 14116 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1a72567fd96847b89eb6e1c4aacaafc1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 03:08:45.047698 14075 catalog_manager.cc:1259] Loaded cluster ID: ea50c45d0fc848439ca73bba6867024a
I20260620 03:08:45.047794 14075 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f: loading cluster ID for follower catalog manager: success
I20260620 03:08:45.050446 14075 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f: acquiring CA information for follower catalog manager: success
I20260620 03:08:45.052246 14075 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 47fa4e000b12460c909ed915cf74228f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 03:08:45.477182 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 13835
I20260620 03:08:45.478374 13897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:45.590052 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 13835
W20260620 03:08:45.611011 14002 connection.cc:535] server connection from 127.0.0.1:34914 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 03:08:45.611757 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 13909
I20260620 03:08:45.612802 13971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:45.718945 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 13909
I20260620 03:08:45.738968 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 13983
I20260620 03:08:45.740202 14045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:45.847214 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 13983
2026-06-20T03:08:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4740 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:08:45 dist-test-slave-xv68 krb5kdc[14131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:08:45 dist-test-slave-xv68 krb5kdc[14131](info): set up 2 sockets
Jun 20 03:08:45 dist-test-slave-xv68 krb5kdc[14131](info): commencing operation
krb5kdc: starting...
W20260620 03:08:46.979795 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:08:47 dist-test-slave-xv68 krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:08:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:08:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:08:47.218899 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45843
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:08:47.556844 14144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:47.557121 14144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:47.557276 14144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:47.564599 14144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:08:47.564707 14144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:47.564790 14144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:47.564898 14144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:08:47.564975 14144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:08:47.575538 14144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:47.576845 14144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:47.578135 14144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:47.590061 14149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:47.590119 14150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:47.590281 14152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:47.590931 14144 server_base.cc:1034] running on GCE node
I20260620 03:08:47.591838 14144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:47.593317 14144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:47.594551 14144 hybrid_clock.cc:648] HybridClock initialized: now 1781924927594496 us; error 55 us; skew 500 ppm
I20260620 03:08:47.594923 14144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:47 dist-test-slave-xv68 krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:47.598270 14144 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:08:47.600071 14144 webserver.cc:460] Webserver started at http://127.0.97.62:35799/ using document root <none> and password file <none>
I20260620 03:08:47.600834 14144 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:47.600993 14144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:47.601326 14144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:08:47.604673 14144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "034ca6b71a84410c87b1c8c88ad5b46b"
format_stamp: "Formatted at 2026-06-20 03:08:47 on dist-test-slave-xv68"
server_key: "bd9f2063a669323900af2bdb10a52170"
server_key_iv: "10c275e873c4bef661730b02f3b00409"
server_key_version: "encryptionkey@0"
I20260620 03:08:47.605484 14144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "034ca6b71a84410c87b1c8c88ad5b46b"
format_stamp: "Formatted at 2026-06-20 03:08:47 on dist-test-slave-xv68"
server_key: "bd9f2063a669323900af2bdb10a52170"
server_key_iv: "10c275e873c4bef661730b02f3b00409"
server_key_version: "encryptionkey@0"
I20260620 03:08:47.611181 14144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 03:08:47.614880 14159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:47.616381 14144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:08:47.616593 14144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "034ca6b71a84410c87b1c8c88ad5b46b"
format_stamp: "Formatted at 2026-06-20 03:08:47 on dist-test-slave-xv68"
server_key: "bd9f2063a669323900af2bdb10a52170"
server_key_iv: "10c275e873c4bef661730b02f3b00409"
server_key_version: "encryptionkey@0"
I20260620 03:08:47.616817 14144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:47.665638 14144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:47.670132 14144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:47.670428 14144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:47.689129 14144 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45843
I20260620 03:08:47.689174 14210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45843 every 8 connection(s)
I20260620 03:08:47.690629 14144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:08:47.692229 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 14144
I20260620 03:08:47.692445 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:08:47.693229 388 external_mini_cluster.cc:1428] Setting key 97b50a498c4318132a8501f13a8f0b5a
I20260620 03:08:47.695341 14211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:08:47.706321 14211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b: Bootstrap starting.
Jun 20 03:08:47 dist-test-slave-xv68 krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:08:47.709724 14211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b: Neither blocks nor log segments found. Creating new log.
I20260620 03:08:47.710882 14211 log.cc:824] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:47.713737 14211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b: No bootstrap required, opened a new log
I20260620 03:08:47.715364 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:47.695183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.0.97.62:45843)
0620 03:08:47.696140 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:47.696152 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:47.696192 (+ 40us) server_negotiation.cc:407] Connection header received
0620 03:08:47.697830 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:47.697852 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:47.698239 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:47.698652 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:47.700237 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:47.701553 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:47.702886 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:47.703662 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:47.703876 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:47.708025 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:47.708040 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:47.708054 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:47.708095 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:47.711268 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:47.712050 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:47.712056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:47.712062 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:47.712223 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:47.712620 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:47.712626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:47.712629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:47.713108 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:47.713414 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:47.713951 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 03:08:47.714357 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":506,"threads_started":1}
I20260620 03:08:47.719064 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:08:47.721971 14211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034ca6b71a84410c87b1c8c88ad5b46b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45843 } }
I20260620 03:08:47.722411 14211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:08:47.722517 14211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 034ca6b71a84410c87b1c8c88ad5b46b, State: Initialized, Role: FOLLOWER
I20260620 03:08:47.723037 14211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034ca6b71a84410c87b1c8c88ad5b46b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45843 } }
I20260620 03:08:47.723243 14211 raft_consensus.cc:388] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:08:47.723402 14211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:08:47.723536 14211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:08:47.725558 14211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034ca6b71a84410c87b1c8c88ad5b46b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45843 } }
I20260620 03:08:47.725991 14211 leader_election.cc:302] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 034ca6b71a84410c87b1c8c88ad5b46b; no voters:
I20260620 03:08:47.726532 14211 leader_election.cc:288] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:08:47.726703 14216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:08:47.727437 14216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b [term 1 LEADER]: Becoming Leader. State: Replica: 034ca6b71a84410c87b1c8c88ad5b46b, State: Running, Role: LEADER
I20260620 03:08:47.727988 14216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034ca6b71a84410c87b1c8c88ad5b46b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45843 } }
I20260620 03:08:47.728654 14211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:08:47.730763 14218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 034ca6b71a84410c87b1c8c88ad5b46b. Latest consensus state: current_term: 1 leader_uuid: "034ca6b71a84410c87b1c8c88ad5b46b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034ca6b71a84410c87b1c8c88ad5b46b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45843 } } }
I20260620 03:08:47.731222 14218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b [sys.catalog]: This master's current role is: LEADER
I20260620 03:08:47.732139 14217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "034ca6b71a84410c87b1c8c88ad5b46b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034ca6b71a84410c87b1c8c88ad5b46b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45843 } } }
I20260620 03:08:47.732587 14217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b [sys.catalog]: This master's current role is: LEADER
I20260620 03:08:47.732909 14224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:08:47.738137 14224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:08:47.746376 14224 catalog_manager.cc:1347] Generated new cluster ID: 3c2f887c0eb745a9b086e277eec2db6d
I20260620 03:08:47.746482 14224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:08:47.762549 14224 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:08:47.764003 14224 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:08:47.771898 14224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 034ca6b71a84410c87b1c8c88ad5b46b: Generated new TSK 0
I20260620 03:08:47.772853 14224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:08:47.831990 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 14144
I20260620 03:08:47.833271 14206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:47.931598 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 14144
2026-06-20T03:08:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2081 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:08:48 dist-test-slave-xv68 krb5kdc[14239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:08:48 dist-test-slave-xv68 krb5kdc[14239](info): set up 2 sockets
Jun 20 03:08:48 dist-test-slave-xv68 krb5kdc[14239](info): commencing operation
krb5kdc: starting...
W20260620 03:08:49.047838 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:08:49 dist-test-slave-xv68 krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:08:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:08:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:08:49.287546 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40015
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:08:49.633390 14252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:49.633663 14252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:49.633797 14252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:49.641112 14252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:08:49.641215 14252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:49.641294 14252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:49.641393 14252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:08:49.641472 14252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:08:49.651222 14252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:49.652532 14252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:49.653786 14252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:49.665472 14260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:49.665496 14258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:49.665498 14257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:49.666350 14252 server_base.cc:1034] running on GCE node
I20260620 03:08:49.667055 14252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:49.668459 14252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:49.669709 14252 hybrid_clock.cc:648] HybridClock initialized: now 1781924929669625 us; error 54 us; skew 500 ppm
I20260620 03:08:49.670037 14252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:49 dist-test-slave-xv68 krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:49.673218 14252 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:08:49.674677 14252 webserver.cc:460] Webserver started at http://127.0.97.62:45385/ using document root <none> and password file <none>
I20260620 03:08:49.675388 14252 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:49.675495 14252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:49.675766 14252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:08:49.679466 14252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "eb503b7f375f4de7bd196b729032587a"
format_stamp: "Formatted at 2026-06-20 03:08:49 on dist-test-slave-xv68"
server_key: "9b5c65b165145a8d0c1ea869ed680f0b"
server_key_iv: "1c9831aa8218046d15e2595c96665f73"
server_key_version: "encryptionkey@0"
I20260620 03:08:49.680291 14252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "eb503b7f375f4de7bd196b729032587a"
format_stamp: "Formatted at 2026-06-20 03:08:49 on dist-test-slave-xv68"
server_key: "9b5c65b165145a8d0c1ea869ed680f0b"
server_key_iv: "1c9831aa8218046d15e2595c96665f73"
server_key_version: "encryptionkey@0"
I20260620 03:08:49.686014 14252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 03:08:49.689553 14267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:49.691002 14252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:08:49.691232 14252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "eb503b7f375f4de7bd196b729032587a"
format_stamp: "Formatted at 2026-06-20 03:08:49 on dist-test-slave-xv68"
server_key: "9b5c65b165145a8d0c1ea869ed680f0b"
server_key_iv: "1c9831aa8218046d15e2595c96665f73"
server_key_version: "encryptionkey@0"
I20260620 03:08:49.691533 14252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:49.705384 14252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:49.709604 14252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:49.709890 14252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:49.728386 14252 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40015
I20260620 03:08:49.728420 14318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40015 every 8 connection(s)
I20260620 03:08:49.729738 14252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:08:49.731773 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 14252
I20260620 03:08:49.731992 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:08:49.732851 388 external_mini_cluster.cc:1428] Setting key b1764f9b4f3e70a726348243c7422521
I20260620 03:08:49.734012 14319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 03:08:49 dist-test-slave-xv68 krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:08:49.748286 14319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a: Bootstrap starting.
I20260620 03:08:49.752429 14319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a: Neither blocks nor log segments found. Creating new log.
I20260620 03:08:49.753679 14319 log.cc:824] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:49.756742 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:49.734799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.0.97.62:40015)
0620 03:08:49.735548 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:49.735559 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:49.735597 (+ 38us) server_negotiation.cc:407] Connection header received
0620 03:08:49.737090 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:49.737111 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:49.737471 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:49.737785 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:49.739560 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:49.740914 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:49.742738 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:49.743495 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:49.743627 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:49.748257 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:49.748273 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:49.748294 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:49.748340 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:49.752182 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:49.752884 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:49.752892 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:49.752902 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:49.753105 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:49.753485 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:49.753493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:49.753496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:49.754186 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:49.754477 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:49.754965 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 03:08:49.755460 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":338,"threads_started":1}
I20260620 03:08:49.757212 14319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a: No bootstrap required, opened a new log
I20260620 03:08:49.765244 14319 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb503b7f375f4de7bd196b729032587a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40015 } }
I20260620 03:08:49.765575 14319 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:08:49.765646 14319 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb503b7f375f4de7bd196b729032587a, State: Initialized, Role: FOLLOWER
I20260620 03:08:49.766110 14319 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb503b7f375f4de7bd196b729032587a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40015 } }
I20260620 03:08:49.766306 14319 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:08:49.766397 14319 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:08:49.766520 14319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:08:49.768683 14319 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb503b7f375f4de7bd196b729032587a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40015 } }
I20260620 03:08:49.769114 14319 leader_election.cc:302] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb503b7f375f4de7bd196b729032587a; no voters:
I20260620 03:08:49.769666 14319 leader_election.cc:288] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:08:49.769832 14324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:08:49.771013 14324 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 1 LEADER]: Becoming Leader. State: Replica: eb503b7f375f4de7bd196b729032587a, State: Running, Role: LEADER
I20260620 03:08:49.771739 14324 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb503b7f375f4de7bd196b729032587a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40015 } }
I20260620 03:08:49.772106 14319 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:08:49.774926 14325 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb503b7f375f4de7bd196b729032587a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb503b7f375f4de7bd196b729032587a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40015 } } }
I20260620 03:08:49.775484 14325 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [sys.catalog]: This master's current role is: LEADER
I20260620 03:08:49.774907 14326 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb503b7f375f4de7bd196b729032587a. Latest consensus state: current_term: 1 leader_uuid: "eb503b7f375f4de7bd196b729032587a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb503b7f375f4de7bd196b729032587a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40015 } } }
I20260620 03:08:49.775956 14326 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [sys.catalog]: This master's current role is: LEADER
I20260620 03:08:49.777774 14333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:08:49.783042 14333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:08:49.791709 14333 catalog_manager.cc:1347] Generated new cluster ID: 6a7536bc4a8244fa891fedfae810cd12
I20260620 03:08:49.791842 14333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:08:49.799921 14333 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:08:49.801200 14333 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:08:49.818038 14333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a: Generated new TSK 0
I20260620 03:08:49.818979 14333 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:08:49.880791 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40015
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:08:50.233645 14345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:50.233932 14345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:50.234025 14345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:50.241319 14345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:50.241509 14345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:50.241704 14345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:08:50.251420 14345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:50.252755 14345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:50.254083 14345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:50.268046 14353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:50.268069 14350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:50.268105 14351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:50.273082 14345 server_base.cc:1034] running on GCE node
I20260620 03:08:50.273916 14345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:50.275098 14345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:50.276427 14345 hybrid_clock.cc:648] HybridClock initialized: now 1781924930276345 us; error 70 us; skew 500 ppm
I20260620 03:08:50.276779 14345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:50 dist-test-slave-xv68 krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:50.280177 14345 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:08:50.281916 14345 webserver.cc:460] Webserver started at http://127.0.97.1:39945/ using document root <none> and password file <none>
I20260620 03:08:50.282632 14345 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:50.282737 14345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:50.282989 14345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:08:50.286126 14345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "10dd91b1b8904053992a493c49c4e750"
format_stamp: "Formatted at 2026-06-20 03:08:50 on dist-test-slave-xv68"
server_key: "c4bc42111673331eeb2b443ff50677e7"
server_key_iv: "8a3f0bce1d23ec82f3c54a5dc778ee32"
server_key_version: "encryptionkey@0"
I20260620 03:08:50.286999 14345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "10dd91b1b8904053992a493c49c4e750"
format_stamp: "Formatted at 2026-06-20 03:08:50 on dist-test-slave-xv68"
server_key: "c4bc42111673331eeb2b443ff50677e7"
server_key_iv: "8a3f0bce1d23ec82f3c54a5dc778ee32"
server_key_version: "encryptionkey@0"
I20260620 03:08:50.292388 14345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 03:08:50.295960 14360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:50.297428 14345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:08:50.297554 14345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "10dd91b1b8904053992a493c49c4e750"
format_stamp: "Formatted at 2026-06-20 03:08:50 on dist-test-slave-xv68"
server_key: "c4bc42111673331eeb2b443ff50677e7"
server_key_iv: "8a3f0bce1d23ec82f3c54a5dc778ee32"
server_key_version: "encryptionkey@0"
I20260620 03:08:50.297732 14345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:50.312233 14345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:50.316215 14345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:50.316459 14345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:50.317341 14345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:08:50.318704 14345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:08:50.318771 14345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:50.318914 14345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:08:50.318985 14345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:50.351814 14345 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45307
I20260620 03:08:50.351861 14472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45307 every 8 connection(s)
I20260620 03:08:50.353140 14345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:08:50.358301 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 14345
I20260620 03:08:50.358541 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:08:50.359191 388 external_mini_cluster.cc:1428] Setting key ee96683b3c591934c1016e15df2c5dcd
Jun 20 03:08:50 dist-test-slave-xv68 krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:08:50.375888 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:50.356357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55085 (local address 127.0.97.62:40015)
0620 03:08:50.356932 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:50.356938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:50.357995 (+ 1057us) server_negotiation.cc:407] Connection header received
0620 03:08:50.360095 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:50.360101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:50.360200 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:50.360315 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:50.361970 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:50.362728 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:50.364384 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:50.364788 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:50.364931 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:50.368906 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:50.368912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:50.368915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:50.368953 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:50.371989 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:50.373089 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:50.373093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:50.373096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:50.373230 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:50.373775 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:50.373779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:50.373781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:50.374068 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:50.374184 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:50.375270 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0620 03:08:50.375581 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":325,"threads_started":1}
I20260620 03:08:50.377314 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:50.357070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:55085)
0620 03:08:50.357836 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:50.357875 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:50.359786 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:50.360550 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:50.360558 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:50.361224 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:50.361800 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:50.361817 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:50.362895 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:50.362899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:50.364228 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:50.364240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:50.365023 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:50.365031 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:50.365288 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:50.365991 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:50.366013 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:50.368644 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:50.372191 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:50.372204 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:50.372227 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:50.372931 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:50.373354 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:50.373363 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:50.373369 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:50.373650 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:50.374207 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:50.374219 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:50.374730 (+ 511us) client_negotiation.cc:769] Sending connection context
0620 03:08:50.375060 (+ 330us) client_negotiation.cc:240] Negotiation successful
0620 03:08:50.375748 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":315,"threads_started":1}
I20260620 03:08:50.379189 14473 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40015
I20260620 03:08:50.379537 14473 heartbeater.cc:464] Registering TS with master...
I20260620 03:08:50.380312 14473 heartbeater.cc:511] Master 127.0.97.62:40015 requested a full tablet report, sending...
I20260620 03:08:50.382561 14284 ts_manager.cc:194] Registered new tserver with Master: 10dd91b1b8904053992a493c49c4e750 (127.0.97.1:45307)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 03:08:50.384706 14284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:08:50.416131 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40015
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:08:50.750828 14479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:50.751084 14479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:50.751178 14479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:50.758392 14479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:50.758496 14479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:50.758616 14479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:08:50.768237 14479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:50.769477 14479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:50.770742 14479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:50.784826 14484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:50.784744 14485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:50.784848 14487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:50.786088 14479 server_base.cc:1034] running on GCE node
I20260620 03:08:50.786729 14479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:50.787896 14479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:50.789151 14479 hybrid_clock.cc:648] HybridClock initialized: now 1781924930789076 us; error 51 us; skew 500 ppm
I20260620 03:08:50.789489 14479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:50 dist-test-slave-xv68 krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:50.793026 14479 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:08:50.794488 14479 webserver.cc:460] Webserver started at http://127.0.97.2:36419/ using document root <none> and password file <none>
I20260620 03:08:50.795202 14479 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:50.795365 14479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:50.795701 14479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:08:50.798667 14479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "1d3153dce87749938bc8a6124cfd40cd"
format_stamp: "Formatted at 2026-06-20 03:08:50 on dist-test-slave-xv68"
server_key: "b5b5424fb4d29ee37f5510c89e33e832"
server_key_iv: "74ef5dac59a0e2414e2637bc5473250b"
server_key_version: "encryptionkey@0"
I20260620 03:08:50.799502 14479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1d3153dce87749938bc8a6124cfd40cd"
format_stamp: "Formatted at 2026-06-20 03:08:50 on dist-test-slave-xv68"
server_key: "b5b5424fb4d29ee37f5510c89e33e832"
server_key_iv: "74ef5dac59a0e2414e2637bc5473250b"
server_key_version: "encryptionkey@0"
I20260620 03:08:50.804945 14479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:08:50.808687 14494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:50.810005 14479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:08:50.810181 14479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "1d3153dce87749938bc8a6124cfd40cd"
format_stamp: "Formatted at 2026-06-20 03:08:50 on dist-test-slave-xv68"
server_key: "b5b5424fb4d29ee37f5510c89e33e832"
server_key_iv: "74ef5dac59a0e2414e2637bc5473250b"
server_key_version: "encryptionkey@0"
I20260620 03:08:50.810388 14479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:50.828020 14479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:50.831976 14479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:50.832252 14479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:50.833194 14479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:08:50.834486 14479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:08:50.834589 14479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:50.834743 14479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:08:50.834815 14479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:50.869033 14479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35549
I20260620 03:08:50.869122 14606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35549 every 8 connection(s)
I20260620 03:08:50.870373 14479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:08:50.879453 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 14479
I20260620 03:08:50.879670 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:08:50.880095 388 external_mini_cluster.cc:1428] Setting key 9f9f68659ef8b4c9557f3ae2b419c218
Jun 20 03:08:50 dist-test-slave-xv68 krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:08:50.891875 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:50.873032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57495 (local address 127.0.97.62:40015)
0620 03:08:50.873246 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:50.873252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:50.874417 (+ 1165us) server_negotiation.cc:407] Connection header received
0620 03:08:50.876838 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:50.876844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:50.876959 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:50.877085 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:50.878595 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:50.879369 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:50.881225 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:50.881702 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:50.881839 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:50.886085 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:50.886090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:50.886093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:50.886132 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:50.888835 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:50.889573 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:50.889579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:50.889584 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:50.889707 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:50.890127 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:50.890132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:50.890136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:50.890418 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:50.890575 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:50.891256 (+ 681us) server_negotiation.cc:299] Negotiation successful
0620 03:08:50.891515 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 03:08:50.892853 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:50.873505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:57495)
0620 03:08:50.874246 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:50.874289 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:50.876315 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:50.877354 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:50.877362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:50.877878 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:50.878420 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:50.878437 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:50.879541 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:50.879547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:50.881076 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:50.881088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:50.881932 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:50.881940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:50.882234 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:50.882996 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:50.883037 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:50.885824 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:50.888995 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:50.889004 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:50.889022 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:50.889441 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:50.889820 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:50.889826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:50.889829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:50.890015 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:50.890547 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:50.890555 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:50.890904 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 03:08:50.891177 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 03:08:50.891732 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":299,"threads_started":1}
I20260620 03:08:50.894397 14607 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40015
I20260620 03:08:50.894716 14607 heartbeater.cc:464] Registering TS with master...
I20260620 03:08:50.895638 14607 heartbeater.cc:511] Master 127.0.97.62:40015 requested a full tablet report, sending...
I20260620 03:08:50.897221 14284 ts_manager.cc:194] Registered new tserver with Master: 1d3153dce87749938bc8a6124cfd40cd (127.0.97.2:35549)
I20260620 03:08:50.898043 14284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57495
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:08:50.932442 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40015
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:08:51.254325 14612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:51.254597 14612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:51.254762 14612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:51.261993 14612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:51.262106 14612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:51.262321 14612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:08:51.272125 14612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:51.273561 14612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:51.274843 14612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:51.288863 14618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:51.288913 14620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:51.289402 14617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:51.289568 14612 server_base.cc:1034] running on GCE node
I20260620 03:08:51.290242 14612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:51.291128 14612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:51.292337 14612 hybrid_clock.cc:648] HybridClock initialized: now 1781924931292304 us; error 69 us; skew 500 ppm
I20260620 03:08:51.292626 14612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:51 dist-test-slave-xv68 krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:51.295933 14612 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:08:51.297377 14612 webserver.cc:460] Webserver started at http://127.0.97.3:32925/ using document root <none> and password file <none>
I20260620 03:08:51.298014 14612 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:51.298110 14612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:51.298342 14612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:08:51.301229 14612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "7c3ef65af3d94972b7a9d346ed319f74"
format_stamp: "Formatted at 2026-06-20 03:08:51 on dist-test-slave-xv68"
server_key: "791db06dc65e46a34cdda92928d82056"
server_key_iv: "4ad25db0bd85d1370782ac148c537c61"
server_key_version: "encryptionkey@0"
I20260620 03:08:51.301988 14612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7c3ef65af3d94972b7a9d346ed319f74"
format_stamp: "Formatted at 2026-06-20 03:08:51 on dist-test-slave-xv68"
server_key: "791db06dc65e46a34cdda92928d82056"
server_key_iv: "4ad25db0bd85d1370782ac148c537c61"
server_key_version: "encryptionkey@0"
I20260620 03:08:51.307057 14612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:08:51.310503 14627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:51.311942 14612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:08:51.312106 14612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "7c3ef65af3d94972b7a9d346ed319f74"
format_stamp: "Formatted at 2026-06-20 03:08:51 on dist-test-slave-xv68"
server_key: "791db06dc65e46a34cdda92928d82056"
server_key_iv: "4ad25db0bd85d1370782ac148c537c61"
server_key_version: "encryptionkey@0"
I20260620 03:08:51.312290 14612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:51.334856 14612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:51.338750 14612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:51.339030 14612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:51.339986 14612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:08:51.341238 14612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:08:51.341308 14612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:51.341499 14612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:08:51.341552 14612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:51.375967 14612 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34945
I20260620 03:08:51.375999 14739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34945 every 8 connection(s)
I20260620 03:08:51.377311 14612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:08:51.384990 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 14612
I20260620 03:08:51.385186 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:08:51.385738 388 external_mini_cluster.cc:1428] Setting key 53379a47ec746c8966f7830302f20a7c
I20260620 03:08:51.388190 14473 heartbeater.cc:503] Master 127.0.97.62:40015 was elected leader, sending a full tablet report...
Jun 20 03:08:51 dist-test-slave-xv68 krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:08:51.400254 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:51.379963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60065 (local address 127.0.97.62:40015)
0620 03:08:51.380145 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:51.380149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:51.381244 (+ 1095us) server_negotiation.cc:407] Connection header received
0620 03:08:51.383069 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:51.383073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:51.383144 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:51.383237 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:51.384751 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.385891 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.388014 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.388693 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.388828 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:51.392377 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:51.392382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:51.392385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:51.392419 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:51.395752 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:51.396890 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:51.396900 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:51.396904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:51.397106 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:51.397681 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:51.397689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:51.397692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:51.398127 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:51.398313 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:51.399421 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0620 03:08:51.399709 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:08:51.401727 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:51.380356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:60065)
0620 03:08:51.381065 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:51.381102 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:51.382802 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:51.383509 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:51.383516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:51.384020 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:51.384576 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:51.384593 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.386042 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.386046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:51.387857 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:51.387872 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.388927 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.388937 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:51.389191 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:51.389958 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:51.389986 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:51.392179 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:51.395912 (+ 3733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:51.395930 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:51.395947 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:51.396753 (+ 806us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:51.397252 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:51.397259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:51.397266 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:51.397541 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:51.398299 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:51.398311 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:51.398828 (+ 517us) client_negotiation.cc:769] Sending connection context
0620 03:08:51.399240 (+ 412us) client_negotiation.cc:240] Negotiation successful
0620 03:08:51.400017 (+ 777us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":269,"threads_started":1}
I20260620 03:08:51.403391 14740 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40015
I20260620 03:08:51.403723 14740 heartbeater.cc:464] Registering TS with master...
I20260620 03:08:51.404620 14740 heartbeater.cc:511] Master 127.0.97.62:40015 requested a full tablet report, sending...
I20260620 03:08:51.405998 14284 ts_manager.cc:194] Registered new tserver with Master: 7c3ef65af3d94972b7a9d346ed319f74 (127.0.97.3:34945)
I20260620 03:08:51.406808 14284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60065
I20260620 03:08:51.413619 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:08:51.430399 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:51.418455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.0.97.62:40015)
0620 03:08:51.418617 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:51.418622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:51.419056 (+ 434us) server_negotiation.cc:407] Connection header received
0620 03:08:51.419199 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:51.419202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:51.419262 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:51.419394 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:51.420730 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.421429 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.422694 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.423078 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.423167 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:51.425609 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:51.425614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:51.425617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:51.425648 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:51.428189 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:51.428820 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:51.428824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:51.428826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:51.428935 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:51.429242 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:51.429245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:51.429247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:51.429558 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:51.429744 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:51.430034 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 03:08:51.430186 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 03:08:51.434412 14284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:08:51.441622 14284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:08:51.475236 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:51.454243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.0.97.3:34945)
0620 03:08:51.464250 (+ 10007us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:51.464259 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:51.466622 (+ 2363us) server_negotiation.cc:407] Connection header received
0620 03:08:51.466716 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:51.466722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:51.466936 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:51.467100 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:08:51.468117 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.469534 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.472259 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.473657 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.473786 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:51.474049 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:51.474677 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 03:08:51.474857 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9821,"thread_start_us":9674,"threads_started":1}
I20260620 03:08:51.475540 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:51.453608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34945 (local address 127.0.0.1:42192)
0620 03:08:51.466368 (+ 12760us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:51.466384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:51.466490 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:51.467431 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:51.467438 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:51.467464 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:08:51.467886 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:51.467896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.469687 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.469693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:51.472099 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:51.472112 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.474243 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.474249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:51.474366 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:51.474477 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 03:08:51.474617 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 03:08:51.474826 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12618,"mutex_wait_us":192,"thread_start_us":12182,"threads_started":1}
I20260620 03:08:51.476042 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:51.453836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38682 (local address 127.0.97.1:45307)
0620 03:08:51.465588 (+ 11752us) spinlock_profiling.cc:243] Waited 1.19 ms on lock 0x60c000080000. stack: 00007f81f3d3c66e 00007f81f3d3c430 00007f81f2dc35ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f8206c1f775 00007f81f3d821d3 00007f81f3d8591b 00007f81f3d85798 00000000009edb7c 00007f81f3d60eae 00007f81f703a6da 00007f81effd061e
0620 03:08:51.465759 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:51.465769 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:51.466374 (+ 605us) server_negotiation.cc:407] Connection header received
0620 03:08:51.466488 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:51.466496 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:51.466763 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:51.466953 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:08:51.468137 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.469685 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.472302 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.473756 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.473875 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:51.474031 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:51.475122 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0620 03:08:51.475312 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9738,"spinlock_wait_cycles":2727808,"thread_start_us":9720,"threads_started":1}
I20260620 03:08:51.476214 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:51.453182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45307 (local address 127.0.0.1:38682)
0620 03:08:51.466044 (+ 12862us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:51.466069 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:51.466240 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:51.467311 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:51.467319 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:51.467360 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:08:51.467875 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:51.467887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.469842 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.469847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:51.472112 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:51.472127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.474197 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.474204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:51.474462 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:51.474498 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 03:08:51.474592 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 03:08:51.474824 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12723,"mutex_wait_us":78,"thread_start_us":11415,"threads_started":1}
I20260620 03:08:51.477540 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:51.453097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35549 (local address 127.0.0.1:41592)
0620 03:08:51.465139 (+ 12042us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:51.465194 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:51.465449 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:51.468997 (+ 3548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:51.469002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:51.469027 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:08:51.469380 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:51.469390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.471360 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.471368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:51.473625 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:51.473640 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.476035 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.476046 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:51.476621 (+ 575us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:51.476658 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 03:08:51.476769 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 03:08:51.476935 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11691,"mutex_wait_us":5,"thread_start_us":12999,"threads_started":1}
I20260620 03:08:51.479723 14408 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3fa08cfcee45c9867422849ef43c62 (DEFAULT_TABLE table=test-table [id=f1bbfca80aa542b59f57187020927496]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:08:51.479722 14675 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3fa08cfcee45c9867422849ef43c62 (DEFAULT_TABLE table=test-table [id=f1bbfca80aa542b59f57187020927496]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:08:51.480333 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:51.453192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41592 (local address 127.0.97.2:35549)
0620 03:08:51.468435 (+ 15243us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:51.468444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:51.468469 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:08:51.468567 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:51.468575 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:51.468789 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:51.468975 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:08:51.469619 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.471173 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.473821 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.475271 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.475411 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:51.475617 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:51.479518 (+ 3901us) server_negotiation.cc:299] Negotiation successful
0620 03:08:51.479916 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15080,"thread_start_us":14955,"threads_started":1}
I20260620 03:08:51.481201 14675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3fa08cfcee45c9867422849ef43c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:08:51.481201 14408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3fa08cfcee45c9867422849ef43c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:08:51.483603 14542 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3fa08cfcee45c9867422849ef43c62 (DEFAULT_TABLE table=test-table [id=f1bbfca80aa542b59f57187020927496]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:08:51.484661 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3fa08cfcee45c9867422849ef43c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:08:51.497306 14758 tablet_bootstrap.cc:492] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd: Bootstrap starting.
I20260620 03:08:51.500268 14758 tablet_bootstrap.cc:654] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd: Neither blocks nor log segments found. Creating new log.
I20260620 03:08:51.501475 14758 log.cc:824] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:51.502333 14759 tablet_bootstrap.cc:492] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74: Bootstrap starting.
I20260620 03:08:51.502344 14760 tablet_bootstrap.cc:492] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750: Bootstrap starting.
I20260620 03:08:51.504683 14758 tablet_bootstrap.cc:492] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd: No bootstrap required, opened a new log
I20260620 03:08:51.504971 14758 ts_tablet_manager.cc:1397] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 03:08:51.505863 14759 tablet_bootstrap.cc:654] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74: Neither blocks nor log segments found. Creating new log.
I20260620 03:08:51.505859 14760 tablet_bootstrap.cc:654] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750: Neither blocks nor log segments found. Creating new log.
I20260620 03:08:51.507067 14760 log.cc:824] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:51.507067 14759 log.cc:824] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:51.510324 14760 tablet_bootstrap.cc:492] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750: No bootstrap required, opened a new log
I20260620 03:08:51.510582 14759 tablet_bootstrap.cc:492] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74: No bootstrap required, opened a new log
I20260620 03:08:51.510600 14760 ts_tablet_manager.cc:1397] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 03:08:51.510834 14759 ts_tablet_manager.cc:1397] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 03:08:51.514973 14758 raft_consensus.cc:348] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } } peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } }
I20260620 03:08:51.515443 14758 raft_consensus.cc:374] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:08:51.515550 14758 raft_consensus.cc:729] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3153dce87749938bc8a6124cfd40cd, State: Initialized, Role: FOLLOWER
I20260620 03:08:51.516161 14758 consensus_queue.cc:260] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } } peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } }
I20260620 03:08:51.521835 14760 raft_consensus.cc:348] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } } peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } }
I20260620 03:08:51.522221 14760 raft_consensus.cc:374] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:08:51.522341 14760 raft_consensus.cc:729] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10dd91b1b8904053992a493c49c4e750, State: Initialized, Role: FOLLOWER
I20260620 03:08:51.522992 14760 consensus_queue.cc:260] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } } peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } }
I20260620 03:08:51.523473 14759 raft_consensus.cc:348] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } } peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } }
I20260620 03:08:51.524003 14759 raft_consensus.cc:374] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:08:51.524134 14759 raft_consensus.cc:729] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3ef65af3d94972b7a9d346ed319f74, State: Initialized, Role: FOLLOWER
I20260620 03:08:51.524834 14760 ts_tablet_manager.cc:1428] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 03:08:51.524816 14759 consensus_queue.cc:260] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } } peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } }
I20260620 03:08:51.525771 14607 heartbeater.cc:503] Master 127.0.97.62:40015 was elected leader, sending a full tablet report...
I20260620 03:08:51.526633 14758 ts_tablet_manager.cc:1428] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd: Time spent starting tablet: real 0.022s user 0.008s sys 0.014s
I20260620 03:08:51.527966 14740 heartbeater.cc:503] Master 127.0.97.62:40015 was elected leader, sending a full tablet report...
I20260620 03:08:51.528594 14759 ts_tablet_manager.cc:1428] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260620 03:08:51.532905 14766 raft_consensus.cc:482] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:08:51.533139 14766 raft_consensus.cc:504] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } } peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } }
I20260620 03:08:51.534654 14766 leader_election.cc:288] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d3153dce87749938bc8a6124cfd40cd (127.0.97.2:35549), 10dd91b1b8904053992a493c49c4e750 (127.0.97.1:45307)
I20260620 03:08:51.540272 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:51.534956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35549 (local address 127.0.97.3:38737)
0620 03:08:51.535431 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:51.535446 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:51.535610 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:51.535933 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:51.535937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:51.535962 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:08:51.536156 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:51.536165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.537116 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.537121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:51.538421 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:51.538432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.539529 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.539534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:51.539723 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:51.539747 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 03:08:51.539846 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 03:08:51.539974 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":24}
I20260620 03:08:51.540257 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:51.535005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38737 (local address 127.0.97.2:35549)
0620 03:08:51.535207 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:51.535213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:51.535523 (+ 310us) server_negotiation.cc:407] Connection header received
0620 03:08:51.535750 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:51.535756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:51.535824 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:51.535918 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:08:51.536284 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.536992 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.538571 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.539381 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.539521 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:51.539609 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:51.539883 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 03:08:51.540002 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:08:51.541056 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:51.535076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39269 (local address 127.0.97.1:45307)
0620 03:08:51.535251 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:51.535255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:51.535967 (+ 712us) server_negotiation.cc:407] Connection header received
0620 03:08:51.536196 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:51.536201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:51.536276 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:51.536369 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:08:51.536868 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.537552 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.539197 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.540051 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.540204 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:51.540305 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:51.540684 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 03:08:51.540804 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 03:08:51.541179 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:51.535091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45307 (local address 127.0.97.3:39269)
0620 03:08:51.535859 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:51.535873 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:51.536064 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:51.536381 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:51.536385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:51.536407 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:08:51.536708 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:51.536717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.537692 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.537697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:51.539044 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:51.539055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.540253 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.540263 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:51.540517 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:51.540548 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 03:08:51.540660 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 03:08:51.540788 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":3,"spinlock_wait_cycles":52224,"thread_start_us":474,"threads_started":1}
I20260620 03:08:51.541359 14562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3fa08cfcee45c9867422849ef43c62" candidate_uuid: "7c3ef65af3d94972b7a9d346ed319f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3153dce87749938bc8a6124cfd40cd" is_pre_election: true
I20260620 03:08:51.541831 14562 raft_consensus.cc:2454] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c3ef65af3d94972b7a9d346ed319f74 in term 0.
I20260620 03:08:51.541958 14428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3fa08cfcee45c9867422849ef43c62" candidate_uuid: "7c3ef65af3d94972b7a9d346ed319f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10dd91b1b8904053992a493c49c4e750" is_pre_election: true
I20260620 03:08:51.542454 14428 raft_consensus.cc:2454] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c3ef65af3d94972b7a9d346ed319f74 in term 0.
I20260620 03:08:51.542460 14631 leader_election.cc:302] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3153dce87749938bc8a6124cfd40cd, 7c3ef65af3d94972b7a9d346ed319f74; no voters:
I20260620 03:08:51.542827 14766 raft_consensus.cc:2781] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:08:51.542976 14766 raft_consensus.cc:482] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:08:51.543100 14766 raft_consensus.cc:3037] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:08:51.545466 14766 raft_consensus.cc:504] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } } peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } }
I20260620 03:08:51.546120 14766 leader_election.cc:288] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [CANDIDATE]: Term 1 election: Requested vote from peers 1d3153dce87749938bc8a6124cfd40cd (127.0.97.2:35549), 10dd91b1b8904053992a493c49c4e750 (127.0.97.1:45307)
I20260620 03:08:51.546504 14562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3fa08cfcee45c9867422849ef43c62" candidate_uuid: "7c3ef65af3d94972b7a9d346ed319f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3153dce87749938bc8a6124cfd40cd"
I20260620 03:08:51.546629 14428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3fa08cfcee45c9867422849ef43c62" candidate_uuid: "7c3ef65af3d94972b7a9d346ed319f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10dd91b1b8904053992a493c49c4e750"
I20260620 03:08:51.546718 14562 raft_consensus.cc:3037] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:08:51.546826 14428 raft_consensus.cc:3037] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:08:51.549067 14428 raft_consensus.cc:2454] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c3ef65af3d94972b7a9d346ed319f74 in term 1.
I20260620 03:08:51.549177 14562 raft_consensus.cc:2454] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c3ef65af3d94972b7a9d346ed319f74 in term 1.
I20260620 03:08:51.549536 14629 leader_election.cc:302] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10dd91b1b8904053992a493c49c4e750, 7c3ef65af3d94972b7a9d346ed319f74; no voters:
I20260620 03:08:51.549865 14766 raft_consensus.cc:2781] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:08:51.550315 14766 raft_consensus.cc:686] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [term 1 LEADER]: Becoming Leader. State: Replica: 7c3ef65af3d94972b7a9d346ed319f74, State: Running, Role: LEADER
I20260620 03:08:51.550813 14766 consensus_queue.cc:237] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } } peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } }
I20260620 03:08:51.555020 14283 catalog_manager.cc:5495] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c3ef65af3d94972b7a9d346ed319f74 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7c3ef65af3d94972b7a9d346ed319f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } health_report { overall_health: HEALTHY } } }
I20260620 03:08:51.579257 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:51.574085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42208 (local address 127.0.97.3:34945)
0620 03:08:51.574275 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:51.574280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:51.574296 (+ 16us) server_negotiation.cc:407] Connection header received
0620 03:08:51.574412 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:51.574416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:51.574502 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:51.574623 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:08:51.575061 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.575865 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.577312 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.577697 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.577793 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:51.578275 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:08:51.578779 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:08:51.578885 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:51.578994 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 03:08:51.579059 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:08:51.587565 14428 raft_consensus.cc:1261] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Refusing update from remote peer 7c3ef65af3d94972b7a9d346ed319f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:08:51.587574 14562 raft_consensus.cc:1261] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd [term 1 FOLLOWER]: Refusing update from remote peer 7c3ef65af3d94972b7a9d346ed319f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:08:51.588654 14768 consensus_queue.cc:1035] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:08:51.588989 14766 consensus_queue.cc:1035] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:08:51.599812 14773 mvcc.cc:205] Tried to move back new op lower bound from 7298764519771947008 to 7298764519636340736. Current Snapshot: MvccSnapshot[applied={T|T < 7298764519771947008}]
I20260620 03:08:51.614308 14282 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:08:51.614604 14282 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:08:51.618463 14282 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a: Sending DeleteTablet for 3 replicas of tablet 9f3fa08cfcee45c9867422849ef43c62
I20260620 03:08:51.619587 14542 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3fa08cfcee45c9867422849ef43c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:08:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41592
I20260620 03:08:51.619668 14408 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3fa08cfcee45c9867422849ef43c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:08:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38682
I20260620 03:08:51.619972 14675 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3fa08cfcee45c9867422849ef43c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:08:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42192
I20260620 03:08:51.620716 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 14345
I20260620 03:08:51.633366 14783 tablet_replica.cc:330] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74: stopping tablet replica
I20260620 03:08:51.633891 14783 raft_consensus.cc:2229] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:08:51.634603 14783 raft_consensus.cc:2258] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:08:51.635461 14782 tablet_replica.cc:330] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750: stopping tablet replica
I20260620 03:08:51.635924 14782 raft_consensus.cc:2229] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:08:51.636447 14782 raft_consensus.cc:2258] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:08:51.638751 14782 ts_tablet_manager.cc:1905] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:08:51.645164 14783 ts_tablet_manager.cc:1905] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:08:51.647004 14782 ts_tablet_manager.cc:1918] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:08:51.647174 14782 log.cc:1196] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/9f3fa08cfcee45c9867422849ef43c62
I20260620 03:08:51.647831 14782 ts_tablet_manager.cc:1939] T 9f3fa08cfcee45c9867422849ef43c62 P 10dd91b1b8904053992a493c49c4e750: Deleting consensus metadata
I20260620 03:08:51.649731 14269 catalog_manager.cc:4841] TS 10dd91b1b8904053992a493c49c4e750 (127.0.97.1:45307): tablet 9f3fa08cfcee45c9867422849ef43c62 (table test-table [id=f1bbfca80aa542b59f57187020927496]) successfully deleted
I20260620 03:08:51.650620 14781 tablet_replica.cc:330] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd: stopping tablet replica
I20260620 03:08:51.651037 14781 raft_consensus.cc:2229] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:08:51.651559 14781 raft_consensus.cc:2258] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:08:51.653235 14783 ts_tablet_manager.cc:1918] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:08:51.653447 14783 log.cc:1196] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/9f3fa08cfcee45c9867422849ef43c62
I20260620 03:08:51.653599 14781 ts_tablet_manager.cc:1905] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:08:51.654085 14783 ts_tablet_manager.cc:1939] T 9f3fa08cfcee45c9867422849ef43c62 P 7c3ef65af3d94972b7a9d346ed319f74: Deleting consensus metadata
I20260620 03:08:51.655862 14268 catalog_manager.cc:4841] TS 7c3ef65af3d94972b7a9d346ed319f74 (127.0.97.3:34945): tablet 9f3fa08cfcee45c9867422849ef43c62 (table test-table [id=f1bbfca80aa542b59f57187020927496]) successfully deleted
Jun 20 03:08:51 dist-test-slave-xv68 krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:08:51.659864 14781 ts_tablet_manager.cc:1918] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:08:51.660020 14781 log.cc:1196] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/9f3fa08cfcee45c9867422849ef43c62
I20260620 03:08:51.660569 14781 ts_tablet_manager.cc:1939] T 9f3fa08cfcee45c9867422849ef43c62 P 1d3153dce87749938bc8a6124cfd40cd: Deleting consensus metadata
I20260620 03:08:51.662339 14271 catalog_manager.cc:4841] TS 1d3153dce87749938bc8a6124cfd40cd (127.0.97.2:35549): tablet 9f3fa08cfcee45c9867422849ef43c62 (table test-table [id=f1bbfca80aa542b59f57187020927496]) successfully deleted
I20260620 03:08:51.665278 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:51.622221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38696 (local address 127.0.97.1:45307)
0620 03:08:51.622446 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:51.622455 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:51.646181 (+ 23726us) server_negotiation.cc:407] Connection header received
0620 03:08:51.646637 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:51.646643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:51.646738 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:51.646815 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:51.648835 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.649863 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.651885 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.652504 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.652605 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:51.658165 (+ 5560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:51.658180 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:51.658197 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:51.658242 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:51.662124 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:51.662928 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:51.662937 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:51.662940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:51.663100 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:51.663560 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:51.663565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:51.663569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:51.664016 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:51.664362 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:51.664640 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 03:08:51.664863 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 03:08:51.666085 14468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:51.756282 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 14345
I20260620 03:08:51.774533 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 14479
Jun 20 03:08:51 dist-test-slave-xv68 krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:08:51.792339 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:51.775377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.0.97.2:35549)
0620 03:08:51.775602 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:51.775608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:51.775631 (+ 23us) server_negotiation.cc:407] Connection header received
0620 03:08:51.775858 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:51.775864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:51.775950 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:51.776072 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:51.777373 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.778491 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.779972 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.780630 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.780750 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:51.785285 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:51.785304 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:51.785324 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:51.785368 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:51.789296 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:51.790045 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:51.790056 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:51.790062 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:51.790222 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:51.790615 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:51.790620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:51.790623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:51.791103 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:51.791432 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:51.791731 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 03:08:51.791958 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 03:08:51.793113 14602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:51.883395 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 14479
I20260620 03:08:51.901072 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 14612
Jun 20 03:08:51 dist-test-slave-xv68 krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:08:51.917241 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:51.901899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42218 (local address 127.0.97.3:34945)
0620 03:08:51.902075 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:51.902081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:51.902167 (+ 86us) server_negotiation.cc:407] Connection header received
0620 03:08:51.902362 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:51.902367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:51.902447 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:51.902573 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:51.903958 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.904804 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.906099 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:51.906558 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:51.906703 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:51.911213 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:51.911223 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:51.911239 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:51.911275 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:51.914262 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:51.914948 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:51.914957 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:51.914967 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:51.915179 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:51.915592 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:51.915599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:51.915608 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:51.916061 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:51.916427 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:51.916643 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 03:08:51.916864 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 03:08:51.917958 14735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:52.041455 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 14612
I20260620 03:08:52.061230 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 14252
I20260620 03:08:52.062453 14314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:52.173293 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 14252
I20260620 03:08:52.196156 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40015
--webserver_interface=127.0.97.62
--webserver_port=45385
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:08:52.522046 14789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:52.522405 14789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:52.522571 14789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:52.533766 14789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:08:52.533870 14789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:52.533919 14789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:52.533975 14789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:08:52.534018 14789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:08:52.543900 14789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:52.545120 14789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:52.546499 14789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:52.569442 14798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:52.569491 14795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:52.569528 14796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:52.570238 14789 server_base.cc:1034] running on GCE node
I20260620 03:08:52.571128 14789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:52.572750 14789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:52.573989 14789 hybrid_clock.cc:648] HybridClock initialized: now 1781924932573936 us; error 44 us; skew 500 ppm
I20260620 03:08:52.574385 14789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:52 dist-test-slave-xv68 krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:52.577984 14789 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:08:52.579720 14789 webserver.cc:460] Webserver started at http://127.0.97.62:45385/ using document root <none> and password file <none>
I20260620 03:08:52.580493 14789 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:52.580670 14789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:52.586601 14789 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 03:08:52.589535 14805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:52.591024 14789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:08:52.591207 14789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "eb503b7f375f4de7bd196b729032587a"
format_stamp: "Formatted at 2026-06-20 03:08:49 on dist-test-slave-xv68"
server_key: "9b5c65b165145a8d0c1ea869ed680f0b"
server_key_iv: "1c9831aa8218046d15e2595c96665f73"
server_key_version: "encryptionkey@0"
I20260620 03:08:52.592343 14789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:52.611081 14789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:52.615828 14789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:52.616112 14789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:52.634723 14789 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40015
I20260620 03:08:52.634698 14856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40015 every 8 connection(s)
I20260620 03:08:52.636294 14789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:08:52.643128 14857 sys_catalog.cc:263] Verifying existing consensus state
I20260620 03:08:52.643652 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 14789
I20260620 03:08:52.644603 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45307
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40015
--webserver_port=39945
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:08:52.648219 14857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a: Bootstrap starting.
I20260620 03:08:52.669435 14857 log.cc:824] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:52.682078 14857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 03:08:52.682719 14857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a: Bootstrap complete.
I20260620 03:08:52.692022 14857 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb503b7f375f4de7bd196b729032587a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40015 } }
I20260620 03:08:52.692952 14857 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb503b7f375f4de7bd196b729032587a, State: Initialized, Role: FOLLOWER
I20260620 03:08:52.693675 14857 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb503b7f375f4de7bd196b729032587a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40015 } }
I20260620 03:08:52.693917 14857 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:08:52.694037 14857 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:08:52.694217 14857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 1 FOLLOWER]: Advancing to term 2
I20260620 03:08:52.698750 14857 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb503b7f375f4de7bd196b729032587a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40015 } }
I20260620 03:08:52.699357 14857 leader_election.cc:302] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb503b7f375f4de7bd196b729032587a; no voters:
I20260620 03:08:52.700014 14857 leader_election.cc:288] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 03:08:52.700160 14861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 03:08:52.701435 14861 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 2 LEADER]: Becoming Leader. State: Replica: eb503b7f375f4de7bd196b729032587a, State: Running, Role: LEADER
I20260620 03:08:52.701990 14861 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb503b7f375f4de7bd196b729032587a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40015 } }
I20260620 03:08:52.702353 14857 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:08:52.705724 14863 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb503b7f375f4de7bd196b729032587a. Latest consensus state: current_term: 2 leader_uuid: "eb503b7f375f4de7bd196b729032587a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb503b7f375f4de7bd196b729032587a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40015 } } }
I20260620 03:08:52.705803 14862 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eb503b7f375f4de7bd196b729032587a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb503b7f375f4de7bd196b729032587a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40015 } } }
I20260620 03:08:52.706251 14862 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [sys.catalog]: This master's current role is: LEADER
I20260620 03:08:52.706307 14863 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [sys.catalog]: This master's current role is: LEADER
I20260620 03:08:52.716506 14866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:08:52.726254 14866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:08:52.728041 14866 catalog_manager.cc:1259] Loaded cluster ID: 6a7536bc4a8244fa891fedfae810cd12
I20260620 03:08:52.728163 14866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:08:52.732071 14866 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 03:08:52.745419 14878 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 03:08:52.750892 14866 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 03:08:52.758234 14866 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:08:52.759632 14866 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a: Loaded TSK: 0
I20260620 03:08:52.760790 14866 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 03:08:53.008833 14859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:53.009156 14859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:53.009333 14859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:53.017122 14859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:53.017274 14859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:53.017447 14859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:08:53.027221 14859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=39945
--tserver_master_addrs=127.0.97.62:40015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:53.028621 14859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:53.029968 14859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:53.044447 14885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:53.044562 14884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:53.045634 14859 server_base.cc:1034] running on GCE node
W20260620 03:08:53.045692 14887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:53.046522 14859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:53.047767 14859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:53.048997 14859 hybrid_clock.cc:648] HybridClock initialized: now 1781924933048934 us; error 56 us; skew 500 ppm
I20260620 03:08:53.049337 14859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:53 dist-test-slave-xv68 krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:53.055963 14859 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:08:53.057649 14859 webserver.cc:460] Webserver started at http://127.0.97.1:39945/ using document root <none> and password file <none>
I20260620 03:08:53.058349 14859 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:53.058492 14859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:53.064183 14859 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 03:08:53.067003 14894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:53.068495 14859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:08:53.068665 14859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "10dd91b1b8904053992a493c49c4e750"
format_stamp: "Formatted at 2026-06-20 03:08:50 on dist-test-slave-xv68"
server_key: "c4bc42111673331eeb2b443ff50677e7"
server_key_iv: "8a3f0bce1d23ec82f3c54a5dc778ee32"
server_key_version: "encryptionkey@0"
I20260620 03:08:53.069689 14859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:53.105472 14859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:53.109545 14859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:53.109829 14859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:53.110816 14859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:08:53.112146 14859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:08:53.112265 14859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:53.112391 14859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:08:53.112488 14859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:53.146474 14859 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45307
I20260620 03:08:53.146528 15006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45307 every 8 connection(s)
I20260620 03:08:53.148110 14859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:08:53.151360 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 14859
I20260620 03:08:53.152177 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35549
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40015
--webserver_port=36419
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:08:53 dist-test-slave-xv68 krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:08:53.176141 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:53.151200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:39085)
0620 03:08:53.152221 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:53.152271 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:53.155015 (+ 2744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:53.156627 (+ 1612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:53.156641 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:53.157458 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:53.158200 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:53.158227 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:53.160206 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:53.160210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:53.161408 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:53.161420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:53.162908 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:53.162913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:53.163100 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:53.163916 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:53.163956 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:53.166631 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:53.171610 (+ 4979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:53.171618 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:53.171632 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:53.172123 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:53.172689 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:53.172692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:53.172694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:53.172871 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:53.173923 (+ 1052us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:53.173929 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:53.174246 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 03:08:53.174542 (+ 296us) client_negotiation.cc:240] Negotiation successful
0620 03:08:53.175043 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":419,"threads_started":1}
I20260620 03:08:53.176939 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:53.151059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39085 (local address 127.0.97.62:40015)
0620 03:08:53.151827 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:53.151843 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:53.152429 (+ 586us) server_negotiation.cc:407] Connection header received
0620 03:08:53.155742 (+ 3313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:53.155757 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:53.156098 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:53.156404 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:53.158444 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:53.160020 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:53.161558 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:53.162669 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:53.162857 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:53.166950 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:53.166963 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:53.166986 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:53.167037 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:53.171438 (+ 4401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:53.172280 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:53.172292 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:53.172302 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:53.172537 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:53.172990 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:53.172996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:53.173001 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:53.173758 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:53.173966 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:53.174776 (+ 810us) server_negotiation.cc:299] Negotiation successful
0620 03:08:53.175318 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":316,"threads_started":1}
I20260620 03:08:53.181530 15007 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40015
I20260620 03:08:53.181896 15007 heartbeater.cc:464] Registering TS with master...
I20260620 03:08:53.182886 15007 heartbeater.cc:511] Master 127.0.97.62:40015 requested a full tablet report, sending...
I20260620 03:08:53.188241 14822 ts_manager.cc:194] Registered new tserver with Master: 10dd91b1b8904053992a493c49c4e750 (127.0.97.1:45307)
I20260620 03:08:53.189922 14822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39085
W20260620 03:08:53.493716 15011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:53.494064 15011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:53.494247 15011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:53.501720 15011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:53.501896 15011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:53.502074 15011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:08:53.511905 15011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36419
--tserver_master_addrs=127.0.97.62:40015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:53.513196 15011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:53.514500 15011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:53.527889 15016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:53.528240 15019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:53.527870 15017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:53.528676 15011 server_base.cc:1034] running on GCE node
I20260620 03:08:53.529449 15011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:53.530514 15011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:53.531730 15011 hybrid_clock.cc:648] HybridClock initialized: now 1781924933531635 us; error 92 us; skew 500 ppm
I20260620 03:08:53.532048 15011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:53 dist-test-slave-xv68 krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:53.534993 15011 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:08:53.536495 15011 webserver.cc:460] Webserver started at http://127.0.97.2:36419/ using document root <none> and password file <none>
I20260620 03:08:53.537156 15011 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:53.537319 15011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:53.542445 15011 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 03:08:53.545094 15026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:53.546340 15011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 03:08:53.546512 15011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "1d3153dce87749938bc8a6124cfd40cd"
format_stamp: "Formatted at 2026-06-20 03:08:50 on dist-test-slave-xv68"
server_key: "b5b5424fb4d29ee37f5510c89e33e832"
server_key_iv: "74ef5dac59a0e2414e2637bc5473250b"
server_key_version: "encryptionkey@0"
I20260620 03:08:53.547488 15011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:53.561510 15011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:53.565748 15011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:53.565974 15011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:53.567006 15011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:08:53.568338 15011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:08:53.568426 15011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:53.568495 15011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:08:53.568550 15011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:53.602715 15011 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35549
I20260620 03:08:53.602773 15138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35549 every 8 connection(s)
I20260620 03:08:53.604319 15011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:08:53.610216 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 15011
I20260620 03:08:53.610819 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34945
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40015
--webserver_port=32925
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:08:53 dist-test-slave-xv68 krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:08:53.626549 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:53.606869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60881 (local address 127.0.97.62:40015)
0620 03:08:53.607051 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:53.607056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:53.608218 (+ 1162us) server_negotiation.cc:407] Connection header received
0620 03:08:53.610012 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:53.610017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:53.610108 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:53.610234 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:53.611901 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:53.612601 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:53.614518 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:53.615329 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:53.615475 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:53.619681 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:53.619690 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:53.619698 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:53.619743 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:53.623244 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:53.624076 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:53.624084 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:53.624091 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:53.624255 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:53.624651 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:53.624659 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:53.624664 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:53.625048 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:53.625217 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:53.625832 (+ 615us) server_negotiation.cc:299] Negotiation successful
0620 03:08:53.626083 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:08:53.627166 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:53.607285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:60881)
0620 03:08:53.608041 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:53.608077 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:53.609744 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:53.610458 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:53.610466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:53.610994 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:53.611728 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:53.611748 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:53.612770 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:53.612775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:53.614335 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:53.614352 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:53.615780 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:53.615790 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:53.616114 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:53.616849 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:53.616881 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:53.619433 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:53.623498 (+ 4065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:53.623505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:53.623516 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:53.623940 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:53.624362 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:53.624365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:53.624367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:53.624525 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:53.625171 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:53.625176 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:53.625460 (+ 284us) client_negotiation.cc:769] Sending connection context
0620 03:08:53.625669 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 03:08:53.626048 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":282,"threads_started":1}
I20260620 03:08:53.628654 15139 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40015
I20260620 03:08:53.629014 15139 heartbeater.cc:464] Registering TS with master...
I20260620 03:08:53.629691 15139 heartbeater.cc:511] Master 127.0.97.62:40015 requested a full tablet report, sending...
I20260620 03:08:53.631150 14822 ts_manager.cc:194] Registered new tserver with Master: 1d3153dce87749938bc8a6124cfd40cd (127.0.97.2:35549)
I20260620 03:08:53.632519 14822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60881
W20260620 03:08:53.937162 15142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:53.937451 15142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:53.937546 15142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:53.944633 15142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:53.944736 15142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:53.944847 15142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:08:53.954682 15142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=32925
--tserver_master_addrs=127.0.97.62:40015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:53.955994 15142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:53.957283 15142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:53.970763 15148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:53.970809 15147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:53.970827 15150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:53.971571 15142 server_base.cc:1034] running on GCE node
I20260620 03:08:53.972133 15142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:53.973012 15142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:53.974241 15142 hybrid_clock.cc:648] HybridClock initialized: now 1781924933974201 us; error 26 us; skew 500 ppm
I20260620 03:08:53.974609 15142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:53 dist-test-slave-xv68 krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:53.977828 15142 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:08:53.979455 15142 webserver.cc:460] Webserver started at http://127.0.97.3:32925/ using document root <none> and password file <none>
I20260620 03:08:53.980139 15142 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:53.980296 15142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:53.985641 15142 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 03:08:53.988449 15157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:53.989799 15142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:08:53.989987 15142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "7c3ef65af3d94972b7a9d346ed319f74"
format_stamp: "Formatted at 2026-06-20 03:08:51 on dist-test-slave-xv68"
server_key: "791db06dc65e46a34cdda92928d82056"
server_key_iv: "4ad25db0bd85d1370782ac148c537c61"
server_key_version: "encryptionkey@0"
I20260620 03:08:53.990999 15142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:54.006639 15142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:54.010672 15142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:54.010954 15142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:54.011950 15142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:08:54.013288 15142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:08:54.013404 15142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:54.013537 15142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:08:54.013633 15142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:54.046702 15142 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34945
I20260620 03:08:54.046787 15269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34945 every 8 connection(s)
I20260620 03:08:54.048184 15142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:08:54.056749 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 15142
Jun 20 03:08:54 dist-test-slave-xv68 krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:08:54.069559 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.050772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57829 (local address 127.0.97.62:40015)
0620 03:08:54.050978 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:54.050982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:54.052154 (+ 1172us) server_negotiation.cc:407] Connection header received
0620 03:08:54.053872 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:54.053877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:54.053961 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:54.054087 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:54.055527 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.056268 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.057991 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.058499 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.058612 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:54.063120 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:54.063125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:54.063128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:54.063160 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:54.066127 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:54.067037 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:54.067041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:54.067044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:54.067191 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:54.067650 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:54.067656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:54.067659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:54.068021 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:54.068195 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:54.068896 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 03:08:54.069188 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 03:08:54.070670 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.051165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:57829)
0620 03:08:54.052015 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:54.052050 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:54.053624 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:54.054300 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:54.054308 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:54.054786 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:54.055373 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:54.055391 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.056451 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.056456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:54.057800 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:54.057811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.058852 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.058859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:54.059165 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:54.059925 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:54.059958 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:54.062913 (+ 2955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:54.066287 (+ 3374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:54.066297 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:54.066317 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:54.066920 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:54.067332 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:54.067339 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:54.067343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:54.067526 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:54.068155 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:54.068161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:54.068479 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 03:08:54.068741 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 03:08:54.069280 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":282,"threads_started":1}
I20260620 03:08:54.072162 15270 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40015
I20260620 03:08:54.072471 15270 heartbeater.cc:464] Registering TS with master...
I20260620 03:08:54.073148 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.057718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54446 (local address 127.0.97.62:40015)
0620 03:08:54.058199 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:54.058204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:54.058559 (+ 355us) server_negotiation.cc:407] Connection header received
0620 03:08:54.058678 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:54.058683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:54.058750 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:54.058900 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:54.060862 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.061786 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.063650 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.064080 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.064232 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:54.067001 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:54.067010 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:54.067017 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:54.067054 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:54.070037 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:54.071075 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:54.071080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:54.071083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:54.071250 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:54.071692 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:54.071699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:54.071703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:54.071973 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:54.072242 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:54.072549 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 03:08:54.072763 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":243,"threads_started":1}
I20260620 03:08:54.073226 15270 heartbeater.cc:511] Master 127.0.97.62:40015 requested a full tablet report, sending...
I20260620 03:08:54.074959 14822 ts_manager.cc:194] Registered new tserver with Master: 7c3ef65af3d94972b7a9d346ed319f74 (127.0.97.3:34945)
I20260620 03:08:54.075981 14822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57829
I20260620 03:08:54.088171 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:08:54.095240 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.089018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.0.97.62:40015)
0620 03:08:54.089283 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:54.089292 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:54.089689 (+ 397us) server_negotiation.cc:407] Connection header received
0620 03:08:54.089961 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:54.089965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:54.090111 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:54.090247 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:08:54.090616 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.091389 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.092950 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.093362 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.093468 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:54.094003 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:08:54.094553 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:08:54.094692 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:54.094848 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 03:08:54.094917 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 03:08:54.096324 14822 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:08:54.103752 14822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:08:54.126410 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.117895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.0.97.2:35549)
0620 03:08:54.119107 (+ 1212us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:54.119114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:54.119143 (+ 29us) server_negotiation.cc:407] Connection header received
0620 03:08:54.119232 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:54.119239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:54.119473 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:54.119676 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:08:54.120297 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.121711 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.124285 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.125264 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.125413 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:54.125655 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:54.125909 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 03:08:54.126074 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":227,"threads_started":1}
I20260620 03:08:54.126521 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.117201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35549 (local address 127.0.0.1:41610)
0620 03:08:54.118775 (+ 1574us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:54.118793 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:54.118899 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:54.119672 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:54.119677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:54.119707 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:08:54.120109 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:54.120121 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.122078 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.122090 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:54.124105 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:54.124119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.125459 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.125465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:54.125654 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:54.125720 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 03:08:54.125867 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 03:08:54.126055 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1408,"mutex_wait_us":19,"thread_start_us":1147,"threads_started":1}
I20260620 03:08:54.127200 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.115778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.0.97.3:34945)
0620 03:08:54.117110 (+ 1332us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:54.117122 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:54.117785 (+ 663us) server_negotiation.cc:407] Connection header received
0620 03:08:54.118687 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:54.118751 (+ 64us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:54.118975 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:54.119144 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:08:54.120039 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.121636 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.124399 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.125773 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.125905 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:54.126104 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:54.126686 (+ 582us) server_negotiation.cc:299] Negotiation successful
0620 03:08:54.126851 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1139,"thread_start_us":992,"threads_started":1}
I20260620 03:08:54.127604 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.115121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.0.97.1:45307)
0620 03:08:54.115780 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:54.115790 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:54.116543 (+ 753us) server_negotiation.cc:407] Connection header received
0620 03:08:54.116934 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:54.116943 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:54.117185 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:54.117378 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:08:54.119427 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.120930 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.123984 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.125447 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.125572 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:54.125765 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:54.127089 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0620 03:08:54.127264 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":1039,"threads_started":1}
I20260620 03:08:54.127991 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.115125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34945 (local address 127.0.0.1:42232)
0620 03:08:54.117655 (+ 2530us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:54.117674 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:54.118471 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:54.119363 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:54.119368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:54.119391 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:08:54.119784 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:54.119799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.121796 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.121802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:54.124159 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:54.124169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.126282 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.126289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:54.126489 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:54.126526 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 03:08:54.127477 (+ 951us) client_negotiation.cc:240] Negotiation successful
0620 03:08:54.127619 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2374,"mutex_wait_us":61,"thread_start_us":2225,"threads_started":1}
I20260620 03:08:54.128432 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.114991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45307 (local address 127.0.0.1:38706)
0620 03:08:54.116410 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:54.116451 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:54.116682 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:54.118383 (+ 1701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:54.118390 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:54.118484 (+ 94us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:08:54.119112 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:54.119126 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.121111 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.121116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:54.123810 (+ 2694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:54.123827 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.125929 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.125935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:54.126882 (+ 947us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:54.126913 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 03:08:54.127866 (+ 953us) client_negotiation.cc:240] Negotiation successful
0620 03:08:54.128025 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"mutex_wait_us":56,"spinlock_wait_cycles":102784,"thread_start_us":978,"threads_started":1}
I20260620 03:08:54.130826 15074 tablet_service.cc:1467] Processing CreateTablet for tablet ab5fb91a1cd84c76b3d22e6f6920d47b (DEFAULT_TABLE table=test-table [id=1728e4869c0247b68a3c2d70d8c30558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:08:54.131101 15205 tablet_service.cc:1467] Processing CreateTablet for tablet ab5fb91a1cd84c76b3d22e6f6920d47b (DEFAULT_TABLE table=test-table [id=1728e4869c0247b68a3c2d70d8c30558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:08:54.132308 15074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab5fb91a1cd84c76b3d22e6f6920d47b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:08:54.132416 15205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab5fb91a1cd84c76b3d22e6f6920d47b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:08:54.132125 14942 tablet_service.cc:1467] Processing CreateTablet for tablet ab5fb91a1cd84c76b3d22e6f6920d47b (DEFAULT_TABLE table=test-table [id=1728e4869c0247b68a3c2d70d8c30558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:08:54.133358 14942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab5fb91a1cd84c76b3d22e6f6920d47b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:08:54.145128 15286 tablet_bootstrap.cc:492] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74: Bootstrap starting.
I20260620 03:08:54.148626 15286 tablet_bootstrap.cc:654] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74: Neither blocks nor log segments found. Creating new log.
I20260620 03:08:54.149967 15286 log.cc:824] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:54.153605 15288 tablet_bootstrap.cc:492] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750: Bootstrap starting.
I20260620 03:08:54.158286 15288 tablet_bootstrap.cc:654] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750: Neither blocks nor log segments found. Creating new log.
I20260620 03:08:54.159960 15288 log.cc:824] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:54.161314 15287 tablet_bootstrap.cc:492] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd: Bootstrap starting.
I20260620 03:08:54.164495 15287 tablet_bootstrap.cc:654] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd: Neither blocks nor log segments found. Creating new log.
I20260620 03:08:54.165731 15287 log.cc:824] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:54.166736 15286 tablet_bootstrap.cc:492] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74: No bootstrap required, opened a new log
I20260620 03:08:54.167068 15286 ts_tablet_manager.cc:1397] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.012s
I20260620 03:08:54.168529 15287 tablet_bootstrap.cc:492] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd: No bootstrap required, opened a new log
I20260620 03:08:54.168802 15287 ts_tablet_manager.cc:1397] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 03:08:54.168926 15288 tablet_bootstrap.cc:492] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750: No bootstrap required, opened a new log
I20260620 03:08:54.169188 15288 ts_tablet_manager.cc:1397] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.005s
I20260620 03:08:54.181612 15286 raft_consensus.cc:348] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } } peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } }
I20260620 03:08:54.182294 15286 raft_consensus.cc:374] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:08:54.182439 15286 raft_consensus.cc:729] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3ef65af3d94972b7a9d346ed319f74, State: Initialized, Role: FOLLOWER
I20260620 03:08:54.183279 15286 consensus_queue.cc:260] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } } peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } }
I20260620 03:08:54.184151 15287 raft_consensus.cc:348] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } } peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } }
I20260620 03:08:54.184695 15287 raft_consensus.cc:374] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:08:54.184828 15287 raft_consensus.cc:729] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3153dce87749938bc8a6124cfd40cd, State: Initialized, Role: FOLLOWER
I20260620 03:08:54.185468 15287 consensus_queue.cc:260] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } } peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } }
I20260620 03:08:54.186619 15139 heartbeater.cc:503] Master 127.0.97.62:40015 was elected leader, sending a full tablet report...
I20260620 03:08:54.187422 15287 ts_tablet_manager.cc:1428] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd: Time spent starting tablet: real 0.018s user 0.014s sys 0.000s
I20260620 03:08:54.188584 15270 heartbeater.cc:503] Master 127.0.97.62:40015 was elected leader, sending a full tablet report...
I20260620 03:08:54.189531 15288 raft_consensus.cc:348] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } } peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } }
I20260620 03:08:54.190043 15288 raft_consensus.cc:374] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:08:54.190171 15288 raft_consensus.cc:729] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10dd91b1b8904053992a493c49c4e750, State: Initialized, Role: FOLLOWER
I20260620 03:08:54.191025 15288 consensus_queue.cc:260] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } } peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } }
I20260620 03:08:54.192445 15007 heartbeater.cc:503] Master 127.0.97.62:40015 was elected leader, sending a full tablet report...
I20260620 03:08:54.193126 15288 ts_tablet_manager.cc:1428] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750: Time spent starting tablet: real 0.024s user 0.016s sys 0.001s
I20260620 03:08:54.193672 15286 ts_tablet_manager.cc:1428] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74: Time spent starting tablet: real 0.026s user 0.017s sys 0.002s
I20260620 03:08:54.253105 15292 raft_consensus.cc:482] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:08:54.253432 15292 raft_consensus.cc:504] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } } peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } }
I20260620 03:08:54.255136 15292 leader_election.cc:288] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10dd91b1b8904053992a493c49c4e750 (127.0.97.1:45307), 1d3153dce87749938bc8a6124cfd40cd (127.0.97.2:35549)
I20260620 03:08:54.262882 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.256556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39023 (local address 127.0.97.1:45307)
0620 03:08:54.256774 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:54.256782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:54.256803 (+ 21us) server_negotiation.cc:407] Connection header received
0620 03:08:54.256875 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:54.256878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:54.256932 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:54.257033 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:08:54.257485 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.258815 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.261108 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.261974 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.262122 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:54.262227 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:54.262490 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 03:08:54.262614 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 03:08:54.262900 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.255677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45307 (local address 127.0.97.3:39023)
0620 03:08:54.256228 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:54.256254 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:54.256423 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:54.257065 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:54.257070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:54.257095 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:08:54.257311 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:54.257328 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.258965 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.258970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:54.260862 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:54.260872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.262144 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.262151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:54.262334 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:54.262358 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 03:08:54.262462 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 03:08:54.262593 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":44}
I20260620 03:08:54.264036 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.255754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35157 (local address 127.0.97.2:35549)
0620 03:08:54.255995 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:54.256003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:54.256644 (+ 641us) server_negotiation.cc:407] Connection header received
0620 03:08:54.256865 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:54.256873 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:54.256972 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:54.257076 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:08:54.257739 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.258405 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.260933 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.262536 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.262680 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:54.262813 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:54.263590 (+ 777us) server_negotiation.cc:299] Negotiation successful
0620 03:08:54.263755 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 03:08:54.264307 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.255742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35549 (local address 127.0.97.3:35157)
0620 03:08:54.256525 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:54.256541 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:54.256707 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:54.257209 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:54.257214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:54.257244 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:08:54.257579 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:54.257586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.258545 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.258550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:54.260749 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:54.260769 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.263130 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.263138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:54.263400 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:54.263450 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 03:08:54.263557 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 03:08:54.263716 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":47,"thread_start_us":481,"threads_started":1}
I20260620 03:08:54.265347 15094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5fb91a1cd84c76b3d22e6f6920d47b" candidate_uuid: "7c3ef65af3d94972b7a9d346ed319f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3153dce87749938bc8a6124cfd40cd" is_pre_election: true
I20260620 03:08:54.265394 14962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5fb91a1cd84c76b3d22e6f6920d47b" candidate_uuid: "7c3ef65af3d94972b7a9d346ed319f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10dd91b1b8904053992a493c49c4e750" is_pre_election: true
I20260620 03:08:54.265884 15094 raft_consensus.cc:2454] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c3ef65af3d94972b7a9d346ed319f74 in term 0.
I20260620 03:08:54.266058 14962 raft_consensus.cc:2454] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c3ef65af3d94972b7a9d346ed319f74 in term 0.
I20260620 03:08:54.266620 15161 leader_election.cc:302] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3153dce87749938bc8a6124cfd40cd, 7c3ef65af3d94972b7a9d346ed319f74; no voters:
I20260620 03:08:54.267263 15292 raft_consensus.cc:2781] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:08:54.267462 15292 raft_consensus.cc:482] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:08:54.267580 15292 raft_consensus.cc:3037] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:08:54.271502 15292 raft_consensus.cc:504] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } } peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } }
I20260620 03:08:54.272406 15292 leader_election.cc:288] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [CANDIDATE]: Term 1 election: Requested vote from peers 10dd91b1b8904053992a493c49c4e750 (127.0.97.1:45307), 1d3153dce87749938bc8a6124cfd40cd (127.0.97.2:35549)
I20260620 03:08:54.272768 14962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5fb91a1cd84c76b3d22e6f6920d47b" candidate_uuid: "7c3ef65af3d94972b7a9d346ed319f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10dd91b1b8904053992a493c49c4e750"
I20260620 03:08:54.272888 15094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5fb91a1cd84c76b3d22e6f6920d47b" candidate_uuid: "7c3ef65af3d94972b7a9d346ed319f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3153dce87749938bc8a6124cfd40cd"
I20260620 03:08:54.273016 14962 raft_consensus.cc:3037] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:08:54.273139 15094 raft_consensus.cc:3037] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:08:54.275581 15094 raft_consensus.cc:2454] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c3ef65af3d94972b7a9d346ed319f74 in term 1.
I20260620 03:08:54.276021 15161 leader_election.cc:302] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3153dce87749938bc8a6124cfd40cd, 7c3ef65af3d94972b7a9d346ed319f74; no voters:
I20260620 03:08:54.276311 15292 raft_consensus.cc:2781] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:08:54.276858 15292 raft_consensus.cc:686] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [term 1 LEADER]: Becoming Leader. State: Replica: 7c3ef65af3d94972b7a9d346ed319f74, State: Running, Role: LEADER
I20260620 03:08:54.277261 14962 raft_consensus.cc:2454] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c3ef65af3d94972b7a9d346ed319f74 in term 1.
I20260620 03:08:54.277382 15292 consensus_queue.cc:237] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } } peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } }
I20260620 03:08:54.282507 14820 catalog_manager.cc:5495] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c3ef65af3d94972b7a9d346ed319f74 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7c3ef65af3d94972b7a9d346ed319f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } health_report { overall_health: UNKNOWN } } }
W20260620 03:08:54.299857 15271 tablet.cc:2369] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:08:54.318415 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.311646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.0.97.3:34945)
0620 03:08:54.311827 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:54.311831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:54.311937 (+ 106us) server_negotiation.cc:407] Connection header received
0620 03:08:54.312176 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:54.312179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:54.312256 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:54.312384 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:08:54.312887 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.313639 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.316077 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.316563 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.316663 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:54.317302 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:08:54.317816 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:08:54.317931 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:54.318077 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 03:08:54.318144 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 03:08:54.326983 14962 raft_consensus.cc:1261] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Refusing update from remote peer 7c3ef65af3d94972b7a9d346ed319f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:08:54.327157 15094 raft_consensus.cc:1261] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd [term 1 FOLLOWER]: Refusing update from remote peer 7c3ef65af3d94972b7a9d346ed319f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:08:54.328213 15292 consensus_queue.cc:1035] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:08:54.328503 15296 consensus_queue.cc:1035] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 03:08:54.356236 15140 tablet.cc:2369] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:08:54.376572 14820 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:08:54.376868 14820 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:08:54.380264 14820 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a: Sending DeleteTablet for 3 replicas of tablet ab5fb91a1cd84c76b3d22e6f6920d47b
I20260620 03:08:54.381592 14942 tablet_service.cc:1514] Processing DeleteTablet for tablet ab5fb91a1cd84c76b3d22e6f6920d47b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:08:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38706
I20260620 03:08:54.381783 15074 tablet_service.cc:1514] Processing DeleteTablet for tablet ab5fb91a1cd84c76b3d22e6f6920d47b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:08:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41610
I20260620 03:08:54.382695 15205 tablet_service.cc:1514] Processing DeleteTablet for tablet ab5fb91a1cd84c76b3d22e6f6920d47b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:08:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42232
I20260620 03:08:54.384253 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 14859
I20260620 03:08:54.408893 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.385126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38708 (local address 127.0.97.1:45307)
0620 03:08:54.385306 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:54.385312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:54.385409 (+ 97us) server_negotiation.cc:407] Connection header received
0620 03:08:54.385605 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:54.385613 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:54.385712 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:54.385821 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:54.387916 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.389526 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.391953 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.392904 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.393032 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:54.397675 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:54.397708 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:54.397733 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:54.397788 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:54.402671 (+ 4883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:54.404108 (+ 1437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:54.404120 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:54.404127 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:54.404309 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:54.405253 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:54.405260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:54.405266 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:54.405884 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:54.406220 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:54.408043 (+ 1823us) server_negotiation.cc:299] Negotiation successful
0620 03:08:54.408341 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 03:08:54.410948 15002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:54.412014 15311 tablet_replica.cc:330] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74: stopping tablet replica
I20260620 03:08:54.412626 15311 raft_consensus.cc:2229] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:08:54.413470 15311 raft_consensus.cc:2258] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:08:54.417496 15311 ts_tablet_manager.cc:1905] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:08:54.426640 15311 ts_tablet_manager.cc:1918] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:08:54.426818 15311 log.cc:1196] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/ab5fb91a1cd84c76b3d22e6f6920d47b
I20260620 03:08:54.427505 15311 ts_tablet_manager.cc:1939] T ab5fb91a1cd84c76b3d22e6f6920d47b P 7c3ef65af3d94972b7a9d346ed319f74: Deleting consensus metadata
I20260620 03:08:54.429661 14806 catalog_manager.cc:4841] TS 7c3ef65af3d94972b7a9d346ed319f74 (127.0.97.3:34945): tablet ab5fb91a1cd84c76b3d22e6f6920d47b (table test-table [id=1728e4869c0247b68a3c2d70d8c30558]) successfully deleted
I20260620 03:08:54.432802 15310 tablet_replica.cc:330] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd: stopping tablet replica
I20260620 03:08:54.433303 15310 raft_consensus.cc:2229] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:08:54.433885 15310 raft_consensus.cc:2258] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:08:54.437019 15310 ts_tablet_manager.cc:1905] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:08:54.444739 15310 ts_tablet_manager.cc:1918] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:08:54.444976 15310 log.cc:1196] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/ab5fb91a1cd84c76b3d22e6f6920d47b
I20260620 03:08:54.445654 15310 ts_tablet_manager.cc:1939] T ab5fb91a1cd84c76b3d22e6f6920d47b P 1d3153dce87749938bc8a6124cfd40cd: Deleting consensus metadata
I20260620 03:08:54.447394 14809 catalog_manager.cc:4841] TS 1d3153dce87749938bc8a6124cfd40cd (127.0.97.2:35549): tablet ab5fb91a1cd84c76b3d22e6f6920d47b (table test-table [id=1728e4869c0247b68a3c2d70d8c30558]) successfully deleted
I20260620 03:08:54.523272 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 14859
I20260620 03:08:54.543318 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 15011
W20260620 03:08:54.543501 14807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45307 (error 108)
I20260620 03:08:54.547252 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.546753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45307 (local address 127.0.0.1:38710)
0620 03:08:54.546962 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:54.547082 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260620 03:08:54.547539 14807 catalog_manager.cc:4584] TS 10dd91b1b8904053992a493c49c4e750 (127.0.97.1:45307): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab5fb91a1cd84c76b3d22e6f6920d47b: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45307: connect: Connection refused (error 111)
I20260620 03:08:54.563002 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.544155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.0.97.2:35549)
0620 03:08:54.544343 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:54.544349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:54.544370 (+ 21us) server_negotiation.cc:407] Connection header received
0620 03:08:54.544600 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:54.544608 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:54.544699 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:54.544817 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:54.546576 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.547662 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.549511 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.549953 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.550085 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:54.556978 (+ 6893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:54.556992 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:54.557008 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:54.557043 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:54.560222 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:54.560946 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:54.560961 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:54.560963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:54.561080 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:54.561463 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:54.561470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:54.561473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:54.561829 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:54.562130 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:54.562441 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 03:08:54.562653 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 03:08:54.563910 15134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:54.601420 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.600884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45307 (local address 127.0.0.1:38720)
0620 03:08:54.601119 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:54.601229 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 03:08:54.602535 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.602145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45307 (local address 127.0.0.1:38730)
0620 03:08:54.602303 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:54.602394 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 03:08:54.663110 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.662515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45307 (local address 127.0.0.1:38742)
0620 03:08:54.662837 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:54.662942 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260620 03:08:54.664809 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.664329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45307 (local address 127.0.0.1:38752)
0620 03:08:54.664560 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:54.664664 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260620 03:08:54.664892 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 15011
I20260620 03:08:54.682925 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 15142
I20260620 03:08:54.699824 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.683757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.0.97.3:34945)
0620 03:08:54.683962 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:54.683968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:54.684046 (+ 78us) server_negotiation.cc:407] Connection header received
0620 03:08:54.684195 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:54.684202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:54.684308 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:54.684442 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:54.685697 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.686934 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.688382 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:54.689098 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:54.689419 (+ 321us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:54.692081 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:54.692101 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:54.692122 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:54.692171 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:54.696620 (+ 4449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:54.697441 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:54.697450 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:54.697454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:54.697659 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:54.698070 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:54.698077 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:54.698082 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:54.698560 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:54.698898 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:54.699109 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 03:08:54.699357 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 03:08:54.700616 15265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:54.745523 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.744960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45307 (local address 127.0.0.1:38766)
0620 03:08:54.745200 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:54.745314 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 03:08:54.746658 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.746238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45307 (local address 127.0.0.1:38776)
0620 03:08:54.746442 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:54.746532 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 03:08:54.825599 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 15142
I20260620 03:08:54.841854 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 14789
I20260620 03:08:54.842967 14852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:08:54.957502 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:54.956682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45307 (local address 127.0.0.1:38778)
0620 03:08:54.956906 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:54.957076 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 03:08:54.957671 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 14789
I20260620 03:08:54.979694 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40015
--webserver_interface=127.0.97.62
--webserver_port=45385
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:08:55.307466 15317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:55.307750 15317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:55.307845 15317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:55.315106 15317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:08:55.315212 15317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:55.315258 15317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:55.315375 15317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:08:55.315428 15317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:08:55.325879 15317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:55.327152 15317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:55.328437 15317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:55.340024 15323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:55.340055 15322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:55.340809 15325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:55.340847 15317 server_base.cc:1034] running on GCE node
I20260620 03:08:55.341689 15317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:55.343084 15317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:55.344326 15317 hybrid_clock.cc:648] HybridClock initialized: now 1781924935344275 us; error 40 us; skew 500 ppm
I20260620 03:08:55.344672 15317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:55 dist-test-slave-xv68 krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:55.347718 15317 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:08:55.349061 15317 webserver.cc:460] Webserver started at http://127.0.97.62:45385/ using document root <none> and password file <none>
I20260620 03:08:55.349659 15317 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:55.349756 15317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:55.355167 15317 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:08:55.357995 15332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:55.359469 15317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:08:55.359647 15317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "eb503b7f375f4de7bd196b729032587a"
format_stamp: "Formatted at 2026-06-20 03:08:49 on dist-test-slave-xv68"
server_key: "9b5c65b165145a8d0c1ea869ed680f0b"
server_key_iv: "1c9831aa8218046d15e2595c96665f73"
server_key_version: "encryptionkey@0"
I20260620 03:08:55.360669 15317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:55.382611 15317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:55.387363 15317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:55.387609 15317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:55.411948 15317 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40015
I20260620 03:08:55.411993 15383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40015 every 8 connection(s)
I20260620 03:08:55.413489 15317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:08:55.420409 15384 sys_catalog.cc:263] Verifying existing consensus state
I20260620 03:08:55.422904 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 15317
I20260620 03:08:55.423722 15384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a: Bootstrap starting.
I20260620 03:08:55.423806 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45307
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40015
--webserver_port=39945
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:08:55.443814 15384 log.cc:824] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:55.460287 15384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 03:08:55.460819 15384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a: Bootstrap complete.
I20260620 03:08:55.470899 15384 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb503b7f375f4de7bd196b729032587a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40015 } }
I20260620 03:08:55.471869 15384 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb503b7f375f4de7bd196b729032587a, State: Initialized, Role: FOLLOWER
I20260620 03:08:55.472535 15384 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb503b7f375f4de7bd196b729032587a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40015 } }
I20260620 03:08:55.472760 15384 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:08:55.472920 15384 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:08:55.473109 15384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 2 FOLLOWER]: Advancing to term 3
I20260620 03:08:55.477088 15384 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb503b7f375f4de7bd196b729032587a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40015 } }
I20260620 03:08:55.477592 15384 leader_election.cc:302] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb503b7f375f4de7bd196b729032587a; no voters:
I20260620 03:08:55.478211 15384 leader_election.cc:288] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 03:08:55.478415 15388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 3 FOLLOWER]: Leader election won for term 3
I20260620 03:08:55.479753 15388 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [term 3 LEADER]: Becoming Leader. State: Replica: eb503b7f375f4de7bd196b729032587a, State: Running, Role: LEADER
I20260620 03:08:55.480321 15388 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb503b7f375f4de7bd196b729032587a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40015 } }
I20260620 03:08:55.480513 15384 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:08:55.483587 15390 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb503b7f375f4de7bd196b729032587a. Latest consensus state: current_term: 3 leader_uuid: "eb503b7f375f4de7bd196b729032587a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb503b7f375f4de7bd196b729032587a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40015 } } }
I20260620 03:08:55.484371 15389 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "eb503b7f375f4de7bd196b729032587a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb503b7f375f4de7bd196b729032587a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40015 } } }
I20260620 03:08:55.484624 15390 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [sys.catalog]: This master's current role is: LEADER
I20260620 03:08:55.484738 15389 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb503b7f375f4de7bd196b729032587a [sys.catalog]: This master's current role is: LEADER
I20260620 03:08:55.496942 15396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:08:55.510226 15396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:08:55.511801 15396 catalog_manager.cc:1259] Loaded cluster ID: 6a7536bc4a8244fa891fedfae810cd12
I20260620 03:08:55.511973 15396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 03:08:55.514057 15396 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2b9c91b9ad google::LogMessage::Fail() at ??:0
@ 0x7f2b9c91e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f2b9c91b4de google::LogMessage::Flush() at ??:0
@ 0x7f2b9c91ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2bb234e512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f2bb234b976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2bb23d3610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f2bb23d3439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2b9f567345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2b9f56a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f2b9f56a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2b9f545eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2ba281f6db start_thread at ??:0
@ 0x7f2b9b7b561f clone at ??:0
W20260620 03:08:55.796757 15386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:55.797111 15386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:55.797282 15386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:55.804821 15386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:55.804979 15386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:55.805150 15386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:08:55.814872 15386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=39945
--tserver_master_addrs=127.0.97.62:40015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:55.816298 15386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:55.817561 15386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:55.829909 15425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:55.830041 15426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:55.830065 15428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:55.831131 15386 server_base.cc:1034] running on GCE node
I20260620 03:08:55.831745 15386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:55.832634 15386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:55.833881 15386 hybrid_clock.cc:648] HybridClock initialized: now 1781924935833829 us; error 44 us; skew 500 ppm
I20260620 03:08:55.834208 15386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:55 dist-test-slave-xv68 krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:55.837355 15386 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:08:55.838773 15386 webserver.cc:460] Webserver started at http://127.0.97.1:39945/ using document root <none> and password file <none>
I20260620 03:08:55.839493 15386 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:55.839636 15386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:55.844643 15386 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:08:55.847396 15435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:55.848686 15386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:08:55.848855 15386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "10dd91b1b8904053992a493c49c4e750"
format_stamp: "Formatted at 2026-06-20 03:08:50 on dist-test-slave-xv68"
server_key: "c4bc42111673331eeb2b443ff50677e7"
server_key_iv: "8a3f0bce1d23ec82f3c54a5dc778ee32"
server_key_version: "encryptionkey@0"
I20260620 03:08:55.849761 15386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:55.875909 15386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:55.880124 15386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:55.880424 15386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:55.881433 15386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:08:55.883368 15442 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 03:08:55.888968 15386 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 03:08:55.889127 15386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 03:08:55.889284 15386 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 03:08:55.892520 15386 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 03:08:55.892629 15386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260620 03:08:55.893097 15442 tablet_bootstrap.cc:492] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750: Bootstrap starting.
I20260620 03:08:55.920441 15442 log.cc:824] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750: Log is configured to *not* fsync() on all Append() calls
I20260620 03:08:55.931203 15386 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45307
I20260620 03:08:55.931226 15549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45307 every 8 connection(s)
I20260620 03:08:55.932621 15386 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:08:55.933825 15442 tablet_bootstrap.cc:492] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 03:08:55.934605 15442 tablet_bootstrap.cc:492] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750: Bootstrap complete.
I20260620 03:08:55.936304 15442 ts_tablet_manager.cc:1397] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750: Time spent bootstrapping tablet: real 0.044s user 0.030s sys 0.011s
I20260620 03:08:55.936990 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 15386
I20260620 03:08:55.937757 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35549
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40015
--webserver_port=36419
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:08:55.938037 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:55.935603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:47161)
0620 03:08:55.936487 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:55.936785 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":352,"threads_started":1}
W20260620 03:08:55.938814 15550 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40015 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40015: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
I20260620 03:08:55.939913 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:55.939459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:38513)
0620 03:08:55.939647 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:55.939749 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 03:08:55.941051 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:55.940586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:43959)
0620 03:08:55.940760 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:55.940874 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 03:08:55.941258 15550 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 03:08:55.943953 15442 raft_consensus.cc:348] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } } peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } }
I20260620 03:08:55.945061 15442 raft_consensus.cc:729] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10dd91b1b8904053992a493c49c4e750, State: Initialized, Role: FOLLOWER
I20260620 03:08:55.945874 15442 consensus_queue.cc:260] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } } peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } }
I20260620 03:08:55.947489 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:55.947100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:37113)
0620 03:08:55.947242 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:55.947371 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 03:08:55.947605 15442 ts_tablet_manager.cc:1428] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260620 03:08:55.948464 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:55.948160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:41193)
0620 03:08:55.948287 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:55.948356 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 03:08:56.283260 15553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:56.283636 15553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:56.283735 15553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:56.290946 15553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:56.291055 15553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:56.291167 15553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:08:56.301107 15553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36419
--tserver_master_addrs=127.0.97.62:40015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:56.302345 15553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:56.303602 15553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:56.316258 15562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:56.316342 15559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:56.316311 15560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:56.317168 15553 server_base.cc:1034] running on GCE node
I20260620 03:08:56.317740 15553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:56.318648 15553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:56.319868 15553 hybrid_clock.cc:648] HybridClock initialized: now 1781924936319829 us; error 53 us; skew 500 ppm
I20260620 03:08:56.320171 15553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:56 dist-test-slave-xv68 krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:56.323146 15553 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:08:56.324563 15553 webserver.cc:460] Webserver started at http://127.0.97.2:36419/ using document root <none> and password file <none>
I20260620 03:08:56.325230 15553 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:56.325380 15553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:56.330529 15553 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 03:08:56.333277 15569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:56.334586 15553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:08:56.334781 15553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "1d3153dce87749938bc8a6124cfd40cd"
format_stamp: "Formatted at 2026-06-20 03:08:50 on dist-test-slave-xv68"
server_key: "b5b5424fb4d29ee37f5510c89e33e832"
server_key_iv: "74ef5dac59a0e2414e2637bc5473250b"
server_key_version: "encryptionkey@0"
I20260620 03:08:56.335776 15553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:56.359078 15553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:56.363255 15553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:56.363617 15553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:56.364609 15553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:08:56.365999 15553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:08:56.366122 15553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:56.366256 15553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:08:56.366350 15553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:56.399927 15553 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35549
I20260620 03:08:56.400013 15681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35549 every 8 connection(s)
I20260620 03:08:56.401350 15553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:08:56.402655 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 15553
I20260620 03:08:56.403468 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34945
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40015
--webserver_port=32925
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:08:56.407013 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:56.404720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:46885)
0620 03:08:56.405578 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:56.405861 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":315,"threads_started":1}
W20260620 03:08:56.407809 15682 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40015 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40015: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
I20260620 03:08:56.408824 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:56.408368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:33109)
0620 03:08:56.408558 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:56.408651 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 03:08:56.409821 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:56.409393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:39755)
0620 03:08:56.409559 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:56.409658 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 03:08:56.410094 15682 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 03:08:56.728396 15685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:08:56.728646 15685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:08:56.728801 15685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:08:56.735988 15685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:08:56.736083 15685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:08:56.736279 15685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:08:56.745805 15685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=32925
--tserver_master_addrs=127.0.97.62:40015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:08:56.747027 15685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:08:56.748327 15685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:08:56.760958 15690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:56.761049 15693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:08:56.761087 15691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:08:56.762108 15685 server_base.cc:1034] running on GCE node
I20260620 03:08:56.762631 15685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:08:56.763525 15685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:08:56.764714 15685 hybrid_clock.cc:648] HybridClock initialized: now 1781924936764664 us; error 73 us; skew 500 ppm
I20260620 03:08:56.765028 15685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:08:56 dist-test-slave-xv68 krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:08:56.768187 15685 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:08:56.769644 15685 webserver.cc:460] Webserver started at http://127.0.97.3:32925/ using document root <none> and password file <none>
I20260620 03:08:56.770316 15685 fs_manager.cc:359] Metadata directory not provided
I20260620 03:08:56.770452 15685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:08:56.775813 15685 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:08:56.778441 15700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:56.779835 15685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:08:56.780014 15685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "7c3ef65af3d94972b7a9d346ed319f74"
format_stamp: "Formatted at 2026-06-20 03:08:51 on dist-test-slave-xv68"
server_key: "791db06dc65e46a34cdda92928d82056"
server_key_iv: "4ad25db0bd85d1370782ac148c537c61"
server_key_version: "encryptionkey@0"
I20260620 03:08:56.780963 15685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:08:56.795639 15685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:08:56.799640 15685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:08:56.799935 15685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:08:56.800907 15685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:08:56.802263 15685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:08:56.802395 15685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:56.802529 15685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:08:56.802620 15685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:08:56.836669 15685 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34945
I20260620 03:08:56.836794 15812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34945 every 8 connection(s)
I20260620 03:08:56.838078 15685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:08:56.842698 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:56.841024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:54255)
0620 03:08:56.841712 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:56.841901 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":278,"threads_started":1}
W20260620 03:08:56.843396 15813 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40015 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40015: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
I20260620 03:08:56.844143 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:56.843855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:39325)
0620 03:08:56.843986 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:56.844048 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 03:08:56.844897 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:56.844594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:38201)
0620 03:08:56.844711 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:56.844791 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 03:08:56.845124 15813 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 03:08:56.847271 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 15685
I20260620 03:08:56.950165 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:56.949345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:42843)
0620 03:08:56.949857 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:56.949960 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":239,"threads_started":1}
I20260620 03:08:57.411501 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:57.410629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:54987)
0620 03:08:57.411104 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:57.411194 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":230,"threads_started":1}
I20260620 03:08:57.786512 15819 raft_consensus.cc:482] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:08:57.786830 15819 raft_consensus.cc:504] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } } peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } }
I20260620 03:08:57.788384 15819 leader_election.cc:288] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c3ef65af3d94972b7a9d346ed319f74 (127.0.97.3:34945), 1d3153dce87749938bc8a6124cfd40cd (127.0.97.2:35549)
Jun 20 03:08:57 dist-test-slave-xv68 krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Jun 20 03:08:57 dist-test-slave-xv68 krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:08:57.813668 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:57.788707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35549 (local address 127.0.97.1:55103)
0620 03:08:57.789390 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:57.789435 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:57.790032 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:57.793702 (+ 3670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:57.793714 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:57.794879 (+ 1165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:57.796059 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:57.796088 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:57.798427 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:57.798431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:57.800802 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:57.800817 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:57.802219 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:57.802227 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:57.802651 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:57.803764 (+ 1113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:57.803787 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:57.806324 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:57.809722 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:57.809733 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:57.809747 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:57.810336 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:57.810836 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:57.810839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:57.810842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:57.811014 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:57.812025 (+ 1011us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:57.812033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:57.812430 (+ 397us) client_negotiation.cc:769] Sending connection context
0620 03:08:57.812717 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 03:08:57.813183 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":345,"spinlock_wait_cycles":17792,"thread_start_us":715,"threads_started":1}
I20260620 03:08:57.813788 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:57.789120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55103 (local address 127.0.97.2:35549)
0620 03:08:57.790078 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:57.790093 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:57.790142 (+ 49us) server_negotiation.cc:407] Connection header received
0620 03:08:57.792615 (+ 2473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:57.792627 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:57.793125 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:57.793512 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:57.796307 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:57.798193 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:57.800963 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:57.801966 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:57.802160 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:57.806637 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:57.806647 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:57.806665 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:57.806713 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:57.809575 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:57.810482 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:57.810488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:57.810493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:57.810713 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:57.811141 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:57.811148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:57.811154 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:57.811888 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:57.812019 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:57.812965 (+ 946us) server_negotiation.cc:299] Negotiation successful
0620 03:08:57.813386 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":645,"threads_started":1}
I20260620 03:08:57.813797 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:57.788751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34945 (local address 127.0.97.1:38247)
0620 03:08:57.789810 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:57.789829 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:08:57.790032 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:08:57.793702 (+ 3670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:08:57.793714 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:08:57.794879 (+ 1165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:08:57.796059 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:57.796089 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:57.798425 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:57.798431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:57.800802 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:08:57.800817 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:08:57.802219 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:08:57.802227 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:08:57.802653 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:57.803795 (+ 1142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:08:57.803821 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:08:57.806686 (+ 2865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:08:57.810579 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:57.810587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:57.810593 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:57.811090 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:57.811542 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:08:57.811547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:08:57.811550 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:08:57.811721 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:08:57.812710 (+ 989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:08:57.812717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:08:57.812937 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 03:08:57.813074 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 03:08:57.813369 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":439,"spinlock_wait_cycles":29440,"thread_start_us":309,"threads_started":1}
I20260620 03:08:57.814165 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:57.789586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38247 (local address 127.0.97.3:34945)
0620 03:08:57.790774 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0620 03:08:57.790788 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 03:08:57.790824 (+ 36us) server_negotiation.cc:407] Connection header received
0620 03:08:57.792942 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:08:57.792953 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:08:57.793327 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:08:57.793640 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:08:57.796307 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:57.798193 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:57.800963 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:08:57.801966 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:08:57.802160 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:08:57.806977 (+ 4817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:08:57.806983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:08:57.806993 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:08:57.807032 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:08:57.810435 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:57.811217 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:57.811223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:57.811228 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:57.811434 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:08:57.811887 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:08:57.811891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:08:57.811893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:08:57.812543 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:08:57.812719 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 03:08:57.813361 (+ 642us) server_negotiation.cc:299] Negotiation successful
0620 03:08:57.813746 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994,"thread_start_us":848,"threads_started":1}
W20260620 03:08:57.817320 15439 leader_election.cc:341] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d3153dce87749938bc8a6124cfd40cd (127.0.97.2:35549): Not found: Tablet not found: ab5fb91a1cd84c76b3d22e6f6920d47b
W20260620 03:08:57.817589 15436 leader_election.cc:341] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c3ef65af3d94972b7a9d346ed319f74 (127.0.97.3:34945): Not found: Tablet not found: ab5fb91a1cd84c76b3d22e6f6920d47b
I20260620 03:08:57.817724 15436 leader_election.cc:302] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10dd91b1b8904053992a493c49c4e750; no voters: 1d3153dce87749938bc8a6124cfd40cd, 7c3ef65af3d94972b7a9d346ed319f74
I20260620 03:08:57.818120 15819 raft_consensus.cc:2726] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 03:08:57.846450 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:57.845624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:34645)
0620 03:08:57.846079 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:57.846205 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":223,"threads_started":1}
I20260620 03:08:57.951478 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:57.951058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:36319)
0620 03:08:57.951230 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:57.951335 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 03:08:58.413045 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:58.412380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:34563)
0620 03:08:58.412797 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:58.412894 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":230,"threads_started":1}
I20260620 03:08:58.848070 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:58.847351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:51739)
0620 03:08:58.847850 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:58.847921 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":232,"threads_started":1}
I20260620 03:08:58.953172 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:58.952349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:40911)
0620 03:08:58.952913 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:58.952992 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":266,"threads_started":1}
I20260620 03:08:59.414717 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:59.413983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:38335)
0620 03:08:59.414443 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:59.414525 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":244,"threads_started":1}
I20260620 03:08:59.849768 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:59.848894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:36315)
0620 03:08:59.849447 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:59.849550 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":324,"threads_started":1}
I20260620 03:08:59.954742 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:08:59.954057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:37419)
0620 03:08:59.954504 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 03:08:59.954586 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":225,"threads_started":1}
I20260620 03:09:00.016903 15831 raft_consensus.cc:482] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:09:00.017089 15831 raft_consensus.cc:504] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } } peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } }
I20260620 03:09:00.017738 15831 leader_election.cc:288] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c3ef65af3d94972b7a9d346ed319f74 (127.0.97.3:34945), 1d3153dce87749938bc8a6124cfd40cd (127.0.97.2:35549)
W20260620 03:09:00.018728 15436 leader_election.cc:341] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c3ef65af3d94972b7a9d346ed319f74 (127.0.97.3:34945): Not found: Tablet not found: ab5fb91a1cd84c76b3d22e6f6920d47b
W20260620 03:09:00.018950 15439 leader_election.cc:341] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d3153dce87749938bc8a6124cfd40cd (127.0.97.2:35549): Not found: Tablet not found: ab5fb91a1cd84c76b3d22e6f6920d47b
I20260620 03:09:00.019163 15439 leader_election.cc:302] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10dd91b1b8904053992a493c49c4e750; no voters: 1d3153dce87749938bc8a6124cfd40cd, 7c3ef65af3d94972b7a9d346ed319f74
I20260620 03:09:00.019426 15831 raft_consensus.cc:2726] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 03:09:00.416473 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:00.415702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:59713)
0620 03:09:00.416178 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:00.416263 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":222,"threads_started":1}
I20260620 03:09:00.851250 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:00.850579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:35401)
0620 03:09:00.851011 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:00.851083 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":220,"threads_started":1}
I20260620 03:09:00.956810 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:00.955985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:36171)
0620 03:09:00.956442 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:00.956564 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":261,"threads_started":1}
I20260620 03:09:01.418094 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:01.417413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:49309)
0620 03:09:01.417851 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:01.417932 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":245,"threads_started":1}
I20260620 03:09:01.852926 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:01.852124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:58653)
0620 03:09:01.852648 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:01.852725 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":316,"threads_started":1}
I20260620 03:09:01.958369 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:01.957659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:59499)
0620 03:09:01.958113 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:01.958201 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":266,"threads_started":1}
I20260620 03:09:02.419878 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:02.419016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:41421)
0620 03:09:02.419612 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:02.419698 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":349,"threads_started":1}
I20260620 03:09:02.854467 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:02.853694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:56161)
0620 03:09:02.854228 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:02.854302 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":339,"threads_started":1}
I20260620 03:09:02.960106 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:02.959264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:35293)
0620 03:09:02.959825 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:02.959909 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":295,"threads_started":1}
I20260620 03:09:03.291873 15841 raft_consensus.cc:482] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:09:03.292147 15841 raft_consensus.cc:504] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } } peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } }
I20260620 03:09:03.292865 15841 leader_election.cc:288] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c3ef65af3d94972b7a9d346ed319f74 (127.0.97.3:34945), 1d3153dce87749938bc8a6124cfd40cd (127.0.97.2:35549)
W20260620 03:09:03.293810 15439 leader_election.cc:341] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d3153dce87749938bc8a6124cfd40cd (127.0.97.2:35549): Not found: Tablet not found: ab5fb91a1cd84c76b3d22e6f6920d47b
W20260620 03:09:03.294008 15436 leader_election.cc:341] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c3ef65af3d94972b7a9d346ed319f74 (127.0.97.3:34945): Not found: Tablet not found: ab5fb91a1cd84c76b3d22e6f6920d47b
I20260620 03:09:03.294144 15436 leader_election.cc:302] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10dd91b1b8904053992a493c49c4e750; no voters: 1d3153dce87749938bc8a6124cfd40cd, 7c3ef65af3d94972b7a9d346ed319f74
I20260620 03:09:03.294345 15841 raft_consensus.cc:2726] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 03:09:03.421675 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:03.420828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:35253)
0620 03:09:03.421393 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:03.421481 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":283,"threads_started":1}
I20260620 03:09:03.856166 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:03.855352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:59821)
0620 03:09:03.855863 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:03.855954 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":217,"threads_started":1}
I20260620 03:09:03.961658 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:03.961012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:56305)
0620 03:09:03.961424 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:03.961502 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":217,"threads_started":1}
I20260620 03:09:04.423415 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:04.422692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:53839)
0620 03:09:04.423097 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:04.423172 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":217,"threads_started":1}
I20260620 03:09:04.857811 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:04.857063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:58959)
0620 03:09:04.857530 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:04.857617 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":243,"threads_started":1}
I20260620 03:09:04.963217 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:04.962477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:54199)
0620 03:09:04.962901 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:04.963013 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":226,"threads_started":1}
I20260620 03:09:05.425103 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:05.424344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:46767)
0620 03:09:05.424819 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:05.424931 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":234,"threads_started":1}
I20260620 03:09:05.859282 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:05.858598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:40499)
0620 03:09:05.859061 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:05.859132 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":219,"threads_started":1}
I20260620 03:09:05.964962 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:05.964288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:40437)
0620 03:09:05.964716 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:05.964796 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":222,"threads_started":1}
I20260620 03:09:06.426841 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:06.426051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:59603)
0620 03:09:06.426587 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:06.426664 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":280,"threads_started":1}
I20260620 03:09:06.604960 15852 raft_consensus.cc:482] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:09:06.605204 15852 raft_consensus.cc:504] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } } peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } }
I20260620 03:09:06.605813 15852 leader_election.cc:288] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c3ef65af3d94972b7a9d346ed319f74 (127.0.97.3:34945), 1d3153dce87749938bc8a6124cfd40cd (127.0.97.2:35549)
W20260620 03:09:06.606726 15436 leader_election.cc:341] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c3ef65af3d94972b7a9d346ed319f74 (127.0.97.3:34945): Not found: Tablet not found: ab5fb91a1cd84c76b3d22e6f6920d47b
W20260620 03:09:06.606926 15439 leader_election.cc:341] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d3153dce87749938bc8a6124cfd40cd (127.0.97.2:35549): Not found: Tablet not found: ab5fb91a1cd84c76b3d22e6f6920d47b
I20260620 03:09:06.607043 15439 leader_election.cc:302] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10dd91b1b8904053992a493c49c4e750; no voters: 1d3153dce87749938bc8a6124cfd40cd, 7c3ef65af3d94972b7a9d346ed319f74
I20260620 03:09:06.607247 15852 raft_consensus.cc:2726] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 03:09:06.860890 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:06.860238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:45357)
0620 03:09:06.860646 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:06.860721 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":210,"threads_started":1}
I20260620 03:09:06.966584 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:06.965822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:35277)
0620 03:09:06.966298 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:06.966381 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":242,"threads_started":1}
I20260620 03:09:07.428709 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:07.427887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:44845)
0620 03:09:07.428461 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:07.428538 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":260,"threads_started":1}
I20260620 03:09:07.862511 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:07.861795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:42787)
0620 03:09:07.862263 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:07.862350 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":236,"threads_started":1}
I20260620 03:09:07.968276 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:07.967578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:49247)
0620 03:09:07.968005 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:07.968084 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":225,"threads_started":1}
I20260620 03:09:08.430399 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:08.429552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:35815)
0620 03:09:08.430053 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:08.430180 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":256,"threads_started":1}
I20260620 03:09:08.864025 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:08.863363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:51953)
0620 03:09:08.863770 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:08.863844 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":210,"threads_started":1}
I20260620 03:09:08.969861 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:08.969101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:42869)
0620 03:09:08.969610 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:08.969691 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":308,"threads_started":1}
I20260620 03:09:09.432101 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:09.431379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:40949)
0620 03:09:09.431832 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:09.431908 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":239,"threads_started":1}
I20260620 03:09:09.865548 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:09.864841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:47347)
0620 03:09:09.865307 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:09.865388 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":268,"threads_started":1}
I20260620 03:09:09.971671 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:09.970725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:40411)
0620 03:09:09.971284 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:09.971431 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":293,"threads_started":1}
I20260620 03:09:10.433612 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:10.432916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:50159)
0620 03:09:10.433351 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:10.433423 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":239,"threads_started":1}
I20260620 03:09:10.649189 15865 raft_consensus.cc:482] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:09:10.649391 15865 raft_consensus.cc:504] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dd91b1b8904053992a493c49c4e750" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45307 } } peers { permanent_uuid: "7c3ef65af3d94972b7a9d346ed319f74" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34945 } } peers { permanent_uuid: "1d3153dce87749938bc8a6124cfd40cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35549 } }
I20260620 03:09:10.650130 15865 leader_election.cc:288] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c3ef65af3d94972b7a9d346ed319f74 (127.0.97.3:34945), 1d3153dce87749938bc8a6124cfd40cd (127.0.97.2:35549)
W20260620 03:09:10.651100 15436 leader_election.cc:341] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c3ef65af3d94972b7a9d346ed319f74 (127.0.97.3:34945): Not found: Tablet not found: ab5fb91a1cd84c76b3d22e6f6920d47b
W20260620 03:09:10.651341 15439 leader_election.cc:341] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d3153dce87749938bc8a6124cfd40cd (127.0.97.2:35549): Not found: Tablet not found: ab5fb91a1cd84c76b3d22e6f6920d47b
I20260620 03:09:10.651489 15439 leader_election.cc:302] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10dd91b1b8904053992a493c49c4e750; no voters: 1d3153dce87749938bc8a6124cfd40cd, 7c3ef65af3d94972b7a9d346ed319f74
I20260620 03:09:10.651736 15865 raft_consensus.cc:2726] T ab5fb91a1cd84c76b3d22e6f6920d47b P 10dd91b1b8904053992a493c49c4e750 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 03:09:10.867028 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:10.866350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:47289)
0620 03:09:10.866776 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:10.866849 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":211,"threads_started":1}
I20260620 03:09:10.973579 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:10.972814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.1:48777)
0620 03:09:10.973298 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:10.973384 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":233,"threads_started":1}
I20260620 03:09:11.435333 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:11.434629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.2:48955)
0620 03:09:11.435038 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:11.435119 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":209,"threads_started":1}
I20260620 03:09:11.858346 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 15386
I20260620 03:09:11.868824 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:11.868069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40015 (local address 127.0.97.3:53043)
0620 03:09:11.868556 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:11.868638 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":273,"threads_started":1}
I20260620 03:09:11.880411 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:11.859818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44628 (local address 127.0.97.1:45307)
0620 03:09:11.860523 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:11.860530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:11.860555 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:09:11.860651 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:11.860658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:11.860948 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:11.861118 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:11.862849 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:11.863921 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:11.865181 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:11.865716 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:11.865849 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:11.874910 (+ 9061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:11.874922 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:11.874936 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:11.874971 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:11.877652 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:11.878353 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:11.878360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:11.878363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:11.878486 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:11.878852 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:11.878859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:11.878862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:11.879212 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:11.879546 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:11.879826 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 03:09:11.880089 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":444,"threads_started":1}
I20260620 03:09:11.881743 15545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:09:11.970259 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 15386
I20260620 03:09:11.988147 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 15553
I20260620 03:09:12.001371 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:11.988864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.0.97.2:35549)
0620 03:09:11.989344 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:11.989348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:11.989365 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:09:11.989420 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:11.989424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:11.989484 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:11.989572 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:11.990880 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:11.991768 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:11.992943 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:11.993359 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:11.993447 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:11.996154 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:11.996161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:11.996164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:11.996206 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:11.998844 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:11.999580 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:11.999585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:11.999587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:11.999726 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:12.000068 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:12.000072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:12.000074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:12.000335 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:12.000550 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:12.000766 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 03:09:12.000935 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":266,"threads_started":1}
I20260620 03:09:12.002130 15677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:09:12.086264 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 15553
I20260620 03:09:12.103703 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 15685
I20260620 03:09:12.117121 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:12.104530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.0.97.3:34945)
0620 03:09:12.105045 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:12.105050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:12.105067 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:09:12.105123 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:12.105127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:12.105217 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:12.105308 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:12.106594 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:12.107463 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:12.108600 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:12.109011 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:12.109108 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:12.112136 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:12.112140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:12.112143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:12.112173 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:12.114767 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:12.115458 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:12.115462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:12.115464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:12.115595 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:12.115908 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:12.115911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:12.115912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:12.116188 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:12.116405 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:12.116584 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 03:09:12.116739 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":251,"threads_started":1}
I20260620 03:09:12.117982 15808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:09:12.202320 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 15685
2026-06-20T03:09:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24271 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:09:12 dist-test-slave-xv68 krb5kdc[15878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:09:12 dist-test-slave-xv68 krb5kdc[15878](info): set up 2 sockets
Jun 20 03:09:12 dist-test-slave-xv68 krb5kdc[15878](info): commencing operation
krb5kdc: starting...
W20260620 03:09:13.312011 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:09:13 dist-test-slave-xv68 krb5kdc[15878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:09:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:09:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:09:13.542287 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38739
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:09:13.867431 15891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:09:13.867764 15891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:09:13.867942 15891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:09:13.875028 15891 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 03:09:13.875167 15891 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 03:09:13.875259 15891 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 03:09:13.875379 15891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:09:13.875460 15891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:09:13.875572 15891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:09:13.875660 15891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:09:13.875741 15891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:09:13.885303 15891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:09:13.886544 15891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:09:13.887858 15891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:09:13.899717 15896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:09:13.899677 15897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:09:13.899739 15899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:09:13.900336 15891 server_base.cc:1034] running on GCE node
I20260620 03:09:13.901070 15891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:09:13.902370 15891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:09:13.903609 15891 hybrid_clock.cc:648] HybridClock initialized: now 1781924953903525 us; error 29 us; skew 500 ppm
I20260620 03:09:13.903941 15891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:09:13 dist-test-slave-xv68 krb5kdc[15878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:09:13.906982 15891 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:09:13.908490 15891 webserver.cc:460] Webserver started at http://127.0.97.62:44111/ using document root <none> and password file <none>
I20260620 03:09:13.909145 15891 fs_manager.cc:359] Metadata directory not provided
I20260620 03:09:13.909255 15891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:09:13.909578 15891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:09:13.912436 15891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "dc34559247524fc28ef40ba4961a7aa0"
format_stamp: "Formatted at 2026-06-20 03:09:13 on dist-test-slave-xv68"
server_key: "261bf68e77cbbbdc47b7f15303eef437"
server_key_iv: "c2d074a369ce2f207b8bdee860ff81f2"
server_key_version: "encryptionkey@0"
I20260620 03:09:13.913180 15891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "dc34559247524fc28ef40ba4961a7aa0"
format_stamp: "Formatted at 2026-06-20 03:09:13 on dist-test-slave-xv68"
server_key: "261bf68e77cbbbdc47b7f15303eef437"
server_key_iv: "c2d074a369ce2f207b8bdee860ff81f2"
server_key_version: "encryptionkey@0"
I20260620 03:09:13.918109 15891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:09:13.921434 15906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:13.922684 15891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 03:09:13.922866 15891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "dc34559247524fc28ef40ba4961a7aa0"
format_stamp: "Formatted at 2026-06-20 03:09:13 on dist-test-slave-xv68"
server_key: "261bf68e77cbbbdc47b7f15303eef437"
server_key_iv: "c2d074a369ce2f207b8bdee860ff81f2"
server_key_version: "encryptionkey@0"
I20260620 03:09:13.923079 15891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:09:13.948673 15891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:09:13.952999 15891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:09:13.953306 15891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:09:13.972033 15891 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38739
I20260620 03:09:13.972052 15957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38739 every 8 connection(s)
I20260620 03:09:13.973356 15891 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:09:13.976979 15958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:09:13.983762 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 15891
I20260620 03:09:13.983974 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:09:13.984589 388 external_mini_cluster.cc:1428] Setting key 0c31dca45de191f66d9ddb7929c4de1d
I20260620 03:09:13.987650 15958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: Bootstrap starting.
I20260620 03:09:13.991093 15958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: Neither blocks nor log segments found. Creating new log.
I20260620 03:09:13.992398 15958 log.cc:824] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: Log is configured to *not* fsync() on all Append() calls
I20260620 03:09:13.995736 15958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: No bootstrap required, opened a new log
Jun 20 03:09:13 dist-test-slave-xv68 krb5kdc[15878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:09:14.004982 15958 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc34559247524fc28ef40ba4961a7aa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38739 } }
I20260620 03:09:14.005388 15958 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:09:14.005509 15958 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc34559247524fc28ef40ba4961a7aa0, State: Initialized, Role: FOLLOWER
I20260620 03:09:14.006220 15958 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc34559247524fc28ef40ba4961a7aa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38739 } }
I20260620 03:09:14.006500 15958 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:09:14.006659 15958 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:09:14.006863 15958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:09:14.009353 15961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:13.986644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39864 (local address 127.0.97.62:38739)
0620 03:09:13.987478 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:13.987489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:13.987530 (+ 41us) server_negotiation.cc:407] Connection header received
0620 03:09:13.989301 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:13.989323 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:13.989687 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:13.990014 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:13.992070 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:13.993854 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:13.995755 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:13.996488 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:13.996624 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:14.001339 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:14.001347 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:14.001361 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:14.001397 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:14.004511 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:14.005193 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:14.005199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:14.005209 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:14.005421 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:14.005926 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:14.005930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:14.005933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:14.006661 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:14.006978 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:14.007504 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 03:09:14.007969 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":378,"threads_started":1}
I20260620 03:09:14.010140 15958 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc34559247524fc28ef40ba4961a7aa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38739 } }
I20260620 03:09:14.010633 15958 leader_election.cc:302] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc34559247524fc28ef40ba4961a7aa0; no voters:
I20260620 03:09:14.011231 15958 leader_election.cc:288] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:09:14.011436 15963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:09:14.013186 15958 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:09:14.013391 15963 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 1 LEADER]: Becoming Leader. State: Replica: dc34559247524fc28ef40ba4961a7aa0, State: Running, Role: LEADER
I20260620 03:09:14.013895 15963 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc34559247524fc28ef40ba4961a7aa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38739 } }
I20260620 03:09:14.016908 15965 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc34559247524fc28ef40ba4961a7aa0. Latest consensus state: current_term: 1 leader_uuid: "dc34559247524fc28ef40ba4961a7aa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc34559247524fc28ef40ba4961a7aa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38739 } } }
I20260620 03:09:14.016935 15964 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc34559247524fc28ef40ba4961a7aa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc34559247524fc28ef40ba4961a7aa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38739 } } }
I20260620 03:09:14.017299 15965 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [sys.catalog]: This master's current role is: LEADER
I20260620 03:09:14.017354 15964 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [sys.catalog]: This master's current role is: LEADER
I20260620 03:09:14.019559 15973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:09:14.024441 15973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:09:14.033993 15973 catalog_manager.cc:1347] Generated new cluster ID: 56bb5e45351a49d884f122283afcd123
I20260620 03:09:14.034121 15973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:09:14.057911 15973 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:09:14.059342 15973 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:09:14.073230 15973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: Generated new TSK 0
I20260620 03:09:14.074000 15973 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:09:14.139045 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38739
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:09:14.471141 15984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:09:14.471480 15984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:09:14.471648 15984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:09:14.478853 15984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:09:14.479007 15984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:09:14.479195 15984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:09:14.488816 15984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:09:14.490088 15984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:09:14.491430 15984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:09:14.504992 15992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:09:14.505064 15989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:09:14.505030 15990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:09:14.508764 15984 server_base.cc:1034] running on GCE node
I20260620 03:09:14.509385 15984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:09:14.510443 15984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:09:14.511663 15984 hybrid_clock.cc:648] HybridClock initialized: now 1781924954511629 us; error 51 us; skew 500 ppm
I20260620 03:09:14.511979 15984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:09:14 dist-test-slave-xv68 krb5kdc[15878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:09:14.515133 15984 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:09:14.517108 15984 webserver.cc:460] Webserver started at http://127.0.97.1:37357/ using document root <none> and password file <none>
I20260620 03:09:14.517834 15984 fs_manager.cc:359] Metadata directory not provided
I20260620 03:09:14.517972 15984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:09:14.518302 15984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:09:14.521246 15984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "deedccf28293476ebd2894c104fb6372"
format_stamp: "Formatted at 2026-06-20 03:09:14 on dist-test-slave-xv68"
server_key: "ba2a3f9b6493c6f14d950ab98e74cd8e"
server_key_iv: "f8829fb2d1300b2302915e106dda1048"
server_key_version: "encryptionkey@0"
I20260620 03:09:14.521989 15984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "deedccf28293476ebd2894c104fb6372"
format_stamp: "Formatted at 2026-06-20 03:09:14 on dist-test-slave-xv68"
server_key: "ba2a3f9b6493c6f14d950ab98e74cd8e"
server_key_iv: "f8829fb2d1300b2302915e106dda1048"
server_key_version: "encryptionkey@0"
I20260620 03:09:14.527096 15984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 03:09:14.530531 15999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:14.531857 15984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:09:14.532032 15984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "deedccf28293476ebd2894c104fb6372"
format_stamp: "Formatted at 2026-06-20 03:09:14 on dist-test-slave-xv68"
server_key: "ba2a3f9b6493c6f14d950ab98e74cd8e"
server_key_iv: "f8829fb2d1300b2302915e106dda1048"
server_key_version: "encryptionkey@0"
I20260620 03:09:14.532234 15984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:09:14.544677 15984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:09:14.548462 15984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:09:14.548743 15984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:09:14.549649 15984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:09:14.550900 15984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:09:14.550973 15984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:14.551136 15984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:09:14.551203 15984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:14.583536 15984 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40843
I20260620 03:09:14.583628 16111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40843 every 8 connection(s)
I20260620 03:09:14.584829 15984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:09:14.594511 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 15984
I20260620 03:09:14.594741 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:09:14.595563 388 external_mini_cluster.cc:1428] Setting key 900015b14eb9ecdb67bf2093a45ee7a4
Jun 20 03:09:14 dist-test-slave-xv68 krb5kdc[15878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:09:14.606850 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:14.587530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49295 (local address 127.0.97.62:38739)
0620 03:09:14.588009 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:14.588013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:14.589035 (+ 1022us) server_negotiation.cc:407] Connection header received
0620 03:09:14.590790 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:14.590795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:14.590866 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:14.590971 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:14.592633 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:14.593460 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:14.595114 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:14.595802 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:14.595969 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:14.600217 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:14.600222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:14.600225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:14.600256 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:14.602792 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:14.603746 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:14.603749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:14.603752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:14.603915 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:14.604457 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:14.604462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:14.604465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:14.604865 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:14.605020 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:14.606142 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0620 03:09:14.606401 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":246,"threads_started":1}
I20260620 03:09:14.607940 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:14.588148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:49295)
0620 03:09:14.588870 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:14.588908 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 03:09:14.590530 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:09:14.591156 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:09:14.591164 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:09:14.591771 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:09:14.592476 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:14.592494 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:14.593609 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:14.593612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:14.594975 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:14.594987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:14.596051 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:14.596057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:14.596296 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:14.597034 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:09:14.597062 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:09:14.600003 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:09:14.602944 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:09:14.602956 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:09:14.602974 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:09:14.603630 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:09:14.604055 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:09:14.604062 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:09:14.604069 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:09:14.604316 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:09:14.605002 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:09:14.605043 (+ 41us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:09:14.605662 (+ 619us) client_negotiation.cc:769] Sending connection context
0620 03:09:14.605990 (+ 328us) client_negotiation.cc:240] Negotiation successful
0620 03:09:14.606544 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":290,"threads_started":1}
I20260620 03:09:14.609438 16112 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38739
I20260620 03:09:14.609786 16112 heartbeater.cc:464] Registering TS with master...
I20260620 03:09:14.610780 16112 heartbeater.cc:511] Master 127.0.97.62:38739 requested a full tablet report, sending...
I20260620 03:09:14.613130 15923 ts_manager.cc:194] Registered new tserver with Master: deedccf28293476ebd2894c104fb6372 (127.0.97.1:40843)
I20260620 03:09:14.614737 15923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49295
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:09:14.652513 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38739
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:09:14.972748 16118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:09:14.973083 16118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:09:14.973268 16118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:09:14.980472 16118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:09:14.980629 16118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:09:14.980829 16118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:09:14.990331 16118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:09:14.991587 16118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:09:14.992890 16118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:09:15.005784 16126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:09:15.005802 16123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:09:15.006453 16124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:09:15.006526 16118 server_base.cc:1034] running on GCE node
I20260620 03:09:15.007095 16118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:09:15.008069 16118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:09:15.009305 16118 hybrid_clock.cc:648] HybridClock initialized: now 1781924955009258 us; error 40 us; skew 500 ppm
I20260620 03:09:15.009616 16118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:09:15 dist-test-slave-xv68 krb5kdc[15878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:09:15.012691 16118 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:09:15.014249 16118 webserver.cc:460] Webserver started at http://127.0.97.2:40109/ using document root <none> and password file <none>
I20260620 03:09:15.014909 16118 fs_manager.cc:359] Metadata directory not provided
I20260620 03:09:15.015053 16118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:09:15.015429 16118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:09:15.018486 16118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "c495ad6c2e444228b91cddf586a37693"
format_stamp: "Formatted at 2026-06-20 03:09:15 on dist-test-slave-xv68"
server_key: "5124765e3bfbda5157b26fc791a49903"
server_key_iv: "2b477e463df3aec3fb93c44376025421"
server_key_version: "encryptionkey@0"
I20260620 03:09:15.019232 16118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c495ad6c2e444228b91cddf586a37693"
format_stamp: "Formatted at 2026-06-20 03:09:15 on dist-test-slave-xv68"
server_key: "5124765e3bfbda5157b26fc791a49903"
server_key_iv: "2b477e463df3aec3fb93c44376025421"
server_key_version: "encryptionkey@0"
I20260620 03:09:15.024274 16118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 03:09:15.027688 16133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:15.029007 16118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 03:09:15.029179 16118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "c495ad6c2e444228b91cddf586a37693"
format_stamp: "Formatted at 2026-06-20 03:09:15 on dist-test-slave-xv68"
server_key: "5124765e3bfbda5157b26fc791a49903"
server_key_iv: "2b477e463df3aec3fb93c44376025421"
server_key_version: "encryptionkey@0"
I20260620 03:09:15.029392 16118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:09:15.048774 16118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:09:15.052615 16118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:09:15.052914 16118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:09:15.053869 16118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:09:15.055156 16118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:09:15.055233 16118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:15.055428 16118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:09:15.055483 16118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:15.086228 16118 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33619
I20260620 03:09:15.086268 16245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33619 every 8 connection(s)
I20260620 03:09:15.087507 16118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:09:15.094056 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 16118
I20260620 03:09:15.094224 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:09:15.094699 388 external_mini_cluster.cc:1428] Setting key 7b0e5c7411d1f07b7d9845edbb8eb329
Jun 20 03:09:15 dist-test-slave-xv68 krb5kdc[15878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:09:15.109243 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:15.090056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49511 (local address 127.0.97.62:38739)
0620 03:09:15.090297 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:15.090302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:15.091359 (+ 1057us) server_negotiation.cc:407] Connection header received
0620 03:09:15.093171 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:15.093175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:15.093241 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:15.093333 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:15.094877 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.095882 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.097947 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.098378 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.098470 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:15.102860 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:15.102865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:15.102868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:15.102901 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:15.106028 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:15.106824 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:15.106830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:15.106833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:15.106974 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:15.107457 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:15.107464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:15.107467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:15.107751 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:15.107920 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:15.108642 (+ 722us) server_negotiation.cc:299] Negotiation successful
0620 03:09:15.108883 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 03:09:15.110210 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:15.090486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:49511)
0620 03:09:15.091169 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:15.091206 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 03:09:15.092847 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:09:15.093530 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:09:15.093538 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:09:15.094023 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:09:15.094658 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:15.094675 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.096018 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.096026 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:15.097786 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:15.097804 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.098609 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.098618 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:15.098944 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:15.099821 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:09:15.099859 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:09:15.102593 (+ 2734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:09:15.106191 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:09:15.106201 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:09:15.106218 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:09:15.106686 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:09:15.107092 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:09:15.107096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:09:15.107100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:09:15.107339 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:09:15.107893 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:09:15.107901 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:09:15.108273 (+ 372us) client_negotiation.cc:769] Sending connection context
0620 03:09:15.108541 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 03:09:15.109091 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":273,"threads_started":1}
I20260620 03:09:15.111755 16246 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38739
I20260620 03:09:15.112072 16246 heartbeater.cc:464] Registering TS with master...
I20260620 03:09:15.112957 16246 heartbeater.cc:511] Master 127.0.97.62:38739 requested a full tablet report, sending...
I20260620 03:09:15.114677 15923 ts_manager.cc:194] Registered new tserver with Master: c495ad6c2e444228b91cddf586a37693 (127.0.97.2:33619)
I20260620 03:09:15.116019 15923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49511
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:09:15.150766 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38739
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:09:15.476215 16251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:09:15.476497 16251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:09:15.476591 16251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:09:15.483665 16251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:09:15.483765 16251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:09:15.483875 16251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:09:15.493156 16251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:09:15.494361 16251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:09:15.495641 16251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:09:15.508352 16259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:09:15.508388 16257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:09:15.508258 16256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:09:15.509555 16251 server_base.cc:1034] running on GCE node
I20260620 03:09:15.510126 16251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:09:15.511127 16251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:09:15.512317 16251 hybrid_clock.cc:648] HybridClock initialized: now 1781924955512270 us; error 45 us; skew 500 ppm
I20260620 03:09:15.512576 16251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:09:15 dist-test-slave-xv68 krb5kdc[15878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:09:15.515779 16251 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:09:15.517284 16251 webserver.cc:460] Webserver started at http://127.0.97.3:43827/ using document root <none> and password file <none>
I20260620 03:09:15.517931 16251 fs_manager.cc:359] Metadata directory not provided
I20260620 03:09:15.518064 16251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:09:15.518464 16251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:09:15.521335 16251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "82eb0c49468442e7b4b6f4029cf899c3"
format_stamp: "Formatted at 2026-06-20 03:09:15 on dist-test-slave-xv68"
server_key: "fb6af908d8a3d3f49a8d1710936d65f5"
server_key_iv: "d02459d083c75280e7378b4ee36405b2"
server_key_version: "encryptionkey@0"
I20260620 03:09:15.522096 16251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "82eb0c49468442e7b4b6f4029cf899c3"
format_stamp: "Formatted at 2026-06-20 03:09:15 on dist-test-slave-xv68"
server_key: "fb6af908d8a3d3f49a8d1710936d65f5"
server_key_iv: "d02459d083c75280e7378b4ee36405b2"
server_key_version: "encryptionkey@0"
I20260620 03:09:15.527244 16251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:09:15.530715 16266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:15.532232 16251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:09:15.532434 16251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "82eb0c49468442e7b4b6f4029cf899c3"
format_stamp: "Formatted at 2026-06-20 03:09:15 on dist-test-slave-xv68"
server_key: "fb6af908d8a3d3f49a8d1710936d65f5"
server_key_iv: "d02459d083c75280e7378b4ee36405b2"
server_key_version: "encryptionkey@0"
I20260620 03:09:15.532630 16251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:09:15.544385 16251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:09:15.548130 16251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:09:15.548414 16251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:09:15.549319 16251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:09:15.550568 16251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:09:15.550640 16251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:15.550788 16251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:09:15.550849 16251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:15.582044 16251 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39969
I20260620 03:09:15.582083 16378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39969 every 8 connection(s)
I20260620 03:09:15.583351 16251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:09:15.592128 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 16251
I20260620 03:09:15.592334 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:09:15.592782 388 external_mini_cluster.cc:1428] Setting key d140d322f289f9deb0a73d3ab9474fdf
Jun 20 03:09:15 dist-test-slave-xv68 krb5kdc[15878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:09:15.604612 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:15.586662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45781 (local address 127.0.97.62:38739)
0620 03:09:15.586892 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:15.586898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:15.588958 (+ 2060us) server_negotiation.cc:407] Connection header received
0620 03:09:15.590731 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:15.590736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:15.590818 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:15.590910 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:15.592532 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.593466 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.595358 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.595854 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.596012 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:15.599366 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:15.599372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:15.599378 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:15.599430 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:15.601961 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:15.602642 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:15.602645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:15.602648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:15.602784 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:15.603153 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:15.603156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:15.603158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:15.603467 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:15.603580 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:15.604216 (+ 636us) server_negotiation.cc:299] Negotiation successful
0620 03:09:15.604376 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 03:09:15.605572 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:15.587116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:45781)
0620 03:09:15.588803 (+ 1687us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:15.588841 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 03:09:15.590475 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:09:15.591075 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:09:15.591082 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:09:15.591696 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:09:15.592323 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:15.592344 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.593608 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.593614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:15.595188 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:15.595206 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.596121 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.596126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:15.596335 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:15.596931 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:09:15.596953 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:09:15.599115 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:09:15.602092 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:09:15.602099 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:09:15.602111 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:09:15.602539 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:09:15.602883 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:09:15.602886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:09:15.602888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:09:15.603062 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:09:15.603573 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:09:15.603579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:09:15.603881 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 03:09:15.604117 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 03:09:15.604500 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1460,"thread_start_us":1267,"threads_started":1}
I20260620 03:09:15.607079 16379 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38739
I20260620 03:09:15.607445 16379 heartbeater.cc:464] Registering TS with master...
I20260620 03:09:15.608330 16379 heartbeater.cc:511] Master 127.0.97.62:38739 requested a full tablet report, sending...
I20260620 03:09:15.610225 15923 ts_manager.cc:194] Registered new tserver with Master: 82eb0c49468442e7b4b6f4029cf899c3 (127.0.97.3:39969)
I20260620 03:09:15.611452 15923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45781
I20260620 03:09:15.618489 16112 heartbeater.cc:503] Master 127.0.97.62:38739 was elected leader, sending a full tablet report...
I20260620 03:09:15.621169 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:09:15.638942 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:15.625623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.0.97.62:38739)
0620 03:09:15.625837 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:15.625845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:15.626271 (+ 426us) server_negotiation.cc:407] Connection header received
0620 03:09:15.626480 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:15.626485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:15.626559 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:15.626669 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:15.628232 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.628954 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.630412 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.630851 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.631001 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:15.633500 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:15.633505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:15.633509 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:15.633540 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:15.636240 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:15.636950 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:15.636955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:15.636959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:15.637081 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:15.637461 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:15.637468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:15.637471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:15.637796 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:15.638062 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:15.638382 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 03:09:15.638597 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 03:09:15.642831 15923 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:09:15.649775 15923 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:09:15.672675 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:15.663748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54708 (local address 127.0.97.2:33619)
0620 03:09:15.665050 (+ 1302us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:15.665060 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:15.665084 (+ 24us) server_negotiation.cc:407] Connection header received
0620 03:09:15.665194 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:15.665201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:15.665394 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:15.665584 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:09:15.666251 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.667878 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.670273 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.671626 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.671776 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:15.671961 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:15.672209 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 03:09:15.672382 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1151,"thread_start_us":1050,"threads_started":1}
I20260620 03:09:15.672693 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:15.663511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33619 (local address 127.0.0.1:54708)
0620 03:09:15.664460 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:15.664477 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:09:15.664575 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:09:15.665589 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:09:15.665594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:09:15.665619 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:09:15.666039 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:15.666055 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.668017 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.668022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:15.670117 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:15.670128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.671847 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.671854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:15.672024 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:15.672048 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 03:09:15.672155 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 03:09:15.672278 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":692,"threads_started":1}
I20260620 03:09:15.672890 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:15.662589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41316 (local address 127.0.97.1:40843)
0620 03:09:15.663723 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:15.663732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:15.663760 (+ 28us) server_negotiation.cc:407] Connection header received
0620 03:09:15.663879 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:15.663888 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:15.664169 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:15.664378 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:09:15.665290 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.666723 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.669671 (+ 2948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.671116 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.671254 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:15.671514 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:15.672413 (+ 899us) server_negotiation.cc:299] Negotiation successful
0620 03:09:15.672572 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":1236,"threads_started":1}
I20260620 03:09:15.673219 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:15.661653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39969 (local address 127.0.0.1:56114)
0620 03:09:15.662704 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:15.662761 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 03:09:15.662991 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:09:15.665829 (+ 2838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:09:15.665834 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:09:15.665857 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:09:15.666639 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:15.666655 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.668433 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.668441 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:15.670352 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:15.670368 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.672386 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.672392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:15.672555 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:15.672583 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 03:09:15.672692 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 03:09:15.672806 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":2,"thread_start_us":564,"threads_started":1}
I20260620 03:09:15.673704 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:15.662069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40843 (local address 127.0.0.1:41316)
0620 03:09:15.663250 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:15.663267 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:09:15.663401 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:09:15.664673 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:09:15.664683 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:09:15.664721 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:09:15.665086 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:15.665099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.666896 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.666901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:15.669500 (+ 2599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:15.669515 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.671636 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.671642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:15.671847 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:15.671901 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 03:09:15.672019 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 03:09:15.672226 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"mutex_wait_us":25,"thread_start_us":902,"threads_started":1}
I20260620 03:09:15.673825 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:15.661957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56114 (local address 127.0.97.3:39969)
0620 03:09:15.665084 (+ 3127us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:15.665091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:15.665120 (+ 29us) server_negotiation.cc:407] Connection header received
0620 03:09:15.665201 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:15.665207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:15.665394 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:15.665580 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:09:15.666837 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.668298 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.670538 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.671918 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.672040 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:15.672238 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:15.673327 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0620 03:09:15.673521 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2968,"thread_start_us":3027,"threads_started":1}
I20260620 03:09:15.676965 16314 tablet_service.cc:1467] Processing CreateTablet for tablet 9f5d020bb94642c5a8ebbba3645791fc (DEFAULT_TABLE table=test-table [id=572811d54a664faeb6d3ffbf2a04bc0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:09:15.677151 16047 tablet_service.cc:1467] Processing CreateTablet for tablet 9f5d020bb94642c5a8ebbba3645791fc (DEFAULT_TABLE table=test-table [id=572811d54a664faeb6d3ffbf2a04bc0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:09:15.677011 16181 tablet_service.cc:1467] Processing CreateTablet for tablet 9f5d020bb94642c5a8ebbba3645791fc (DEFAULT_TABLE table=test-table [id=572811d54a664faeb6d3ffbf2a04bc0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:09:15.678241 16314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f5d020bb94642c5a8ebbba3645791fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:09:15.678288 16047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f5d020bb94642c5a8ebbba3645791fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:09:15.678311 16181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f5d020bb94642c5a8ebbba3645791fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:09:15.689368 16397 tablet_bootstrap.cc:492] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3: Bootstrap starting.
I20260620 03:09:15.692463 16398 tablet_bootstrap.cc:492] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372: Bootstrap starting.
I20260620 03:09:15.693463 16399 tablet_bootstrap.cc:492] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693: Bootstrap starting.
I20260620 03:09:15.694646 16397 tablet_bootstrap.cc:654] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3: Neither blocks nor log segments found. Creating new log.
I20260620 03:09:15.696026 16397 log.cc:824] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3: Log is configured to *not* fsync() on all Append() calls
I20260620 03:09:15.696427 16399 tablet_bootstrap.cc:654] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693: Neither blocks nor log segments found. Creating new log.
I20260620 03:09:15.696604 16398 tablet_bootstrap.cc:654] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372: Neither blocks nor log segments found. Creating new log.
I20260620 03:09:15.697544 16399 log.cc:824] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693: Log is configured to *not* fsync() on all Append() calls
I20260620 03:09:15.698040 16398 log.cc:824] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372: Log is configured to *not* fsync() on all Append() calls
I20260620 03:09:15.700320 16399 tablet_bootstrap.cc:492] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693: No bootstrap required, opened a new log
I20260620 03:09:15.700671 16398 tablet_bootstrap.cc:492] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372: No bootstrap required, opened a new log
I20260620 03:09:15.700667 16399 ts_tablet_manager.cc:1397] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260620 03:09:15.701033 16398 ts_tablet_manager.cc:1397] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 03:09:15.702436 16397 tablet_bootstrap.cc:492] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3: No bootstrap required, opened a new log
I20260620 03:09:15.702697 16397 ts_tablet_manager.cc:1397] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260620 03:09:15.708741 16399 raft_consensus.cc:348] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } } peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } }
I20260620 03:09:15.709124 16399 raft_consensus.cc:374] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:09:15.709242 16399 raft_consensus.cc:729] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c495ad6c2e444228b91cddf586a37693, State: Initialized, Role: FOLLOWER
I20260620 03:09:15.709861 16399 consensus_queue.cc:260] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } } peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } }
I20260620 03:09:15.711006 16246 heartbeater.cc:503] Master 127.0.97.62:38739 was elected leader, sending a full tablet report...
I20260620 03:09:15.711673 16399 ts_tablet_manager.cc:1428] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260620 03:09:15.712479 16398 raft_consensus.cc:348] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } } peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } }
I20260620 03:09:15.713016 16398 raft_consensus.cc:374] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:09:15.713135 16398 raft_consensus.cc:729] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deedccf28293476ebd2894c104fb6372, State: Initialized, Role: FOLLOWER
I20260620 03:09:15.713218 16397 raft_consensus.cc:348] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } } peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } }
I20260620 03:09:15.713769 16397 raft_consensus.cc:374] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:09:15.713877 16397 raft_consensus.cc:729] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82eb0c49468442e7b4b6f4029cf899c3, State: Initialized, Role: FOLLOWER
I20260620 03:09:15.713807 16398 consensus_queue.cc:260] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } } peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } }
I20260620 03:09:15.714594 16397 consensus_queue.cc:260] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } } peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } }
I20260620 03:09:15.715924 16379 heartbeater.cc:503] Master 127.0.97.62:38739 was elected leader, sending a full tablet report...
I20260620 03:09:15.716516 16397 ts_tablet_manager.cc:1428] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 03:09:15.718145 16398 ts_tablet_manager.cc:1428] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
W20260620 03:09:15.836377 16380 tablet.cc:2369] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:09:15.837225 16113 tablet.cc:2369] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:09:15.839498 16247 tablet.cc:2369] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:09:15.970487 16405 raft_consensus.cc:482] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:09:15.970854 16405 raft_consensus.cc:504] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } } peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } }
I20260620 03:09:15.972738 16405 leader_election.cc:288] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers deedccf28293476ebd2894c104fb6372 (127.0.97.1:40843), c495ad6c2e444228b91cddf586a37693 (127.0.97.2:33619)
I20260620 03:09:15.979905 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:15.973216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34505 (local address 127.0.97.1:40843)
0620 03:09:15.973433 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:15.973442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:15.973473 (+ 31us) server_negotiation.cc:407] Connection header received
0620 03:09:15.973756 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:15.973765 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:15.973877 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:15.974000 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:09:15.974667 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.975701 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.978147 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.978955 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.979104 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:15.979206 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:15.979523 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 03:09:15.979661 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 03:09:15.979950 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:15.973044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40843 (local address 127.0.97.3:34505)
0620 03:09:15.973394 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:15.973416 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:09:15.973565 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:09:15.974208 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:09:15.974214 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:09:15.974248 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:09:15.974510 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:15.974520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.975840 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.975848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:15.977979 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:15.977996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.979126 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.979132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:15.979368 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:15.979392 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 03:09:15.979496 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 03:09:15.979643 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":54}
I20260620 03:09:15.980587 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:15.973814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53093 (local address 127.0.97.2:33619)
0620 03:09:15.973999 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:15.974005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:15.974612 (+ 607us) server_negotiation.cc:407] Connection header received
0620 03:09:15.974858 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:15.974864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:15.974955 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:15.975050 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:09:15.975667 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.976426 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.978203 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.979483 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.979615 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:15.979705 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:15.980204 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 03:09:15.980351 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:09:15.981251 16406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:15.973553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33619 (local address 127.0.97.3:53093)
0620 03:09:15.974522 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:15.974538 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:09:15.974703 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:09:15.975098 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:09:15.975105 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:09:15.975134 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:09:15.975508 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:15.975519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.976586 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.976591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:15.978034 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:15.978047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:15.979841 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:15.979848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:15.980030 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:15.980062 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 03:09:15.980752 (+ 690us) client_negotiation.cc:240] Negotiation successful
0620 03:09:15.980883 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":48,"thread_start_us":591,"threads_started":1}
I20260620 03:09:15.981146 16067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5d020bb94642c5a8ebbba3645791fc" candidate_uuid: "82eb0c49468442e7b4b6f4029cf899c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deedccf28293476ebd2894c104fb6372" is_pre_election: true
I20260620 03:09:15.981765 16067 raft_consensus.cc:2454] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82eb0c49468442e7b4b6f4029cf899c3 in term 0.
I20260620 03:09:15.982242 16201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5d020bb94642c5a8ebbba3645791fc" candidate_uuid: "82eb0c49468442e7b4b6f4029cf899c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c495ad6c2e444228b91cddf586a37693" is_pre_election: true
I20260620 03:09:15.982560 16267 leader_election.cc:302] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82eb0c49468442e7b4b6f4029cf899c3, deedccf28293476ebd2894c104fb6372; no voters:
I20260620 03:09:15.982776 16201 raft_consensus.cc:2454] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82eb0c49468442e7b4b6f4029cf899c3 in term 0.
I20260620 03:09:15.982960 16405 raft_consensus.cc:2781] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:09:15.983114 16405 raft_consensus.cc:482] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:09:15.983228 16405 raft_consensus.cc:3037] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:09:15.985531 16405 raft_consensus.cc:504] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } } peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } }
I20260620 03:09:15.986135 16405 leader_election.cc:288] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [CANDIDATE]: Term 1 election: Requested vote from peers deedccf28293476ebd2894c104fb6372 (127.0.97.1:40843), c495ad6c2e444228b91cddf586a37693 (127.0.97.2:33619)
I20260620 03:09:15.986660 16201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5d020bb94642c5a8ebbba3645791fc" candidate_uuid: "82eb0c49468442e7b4b6f4029cf899c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c495ad6c2e444228b91cddf586a37693"
I20260620 03:09:15.986685 16067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5d020bb94642c5a8ebbba3645791fc" candidate_uuid: "82eb0c49468442e7b4b6f4029cf899c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deedccf28293476ebd2894c104fb6372"
I20260620 03:09:15.986876 16201 raft_consensus.cc:3037] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:09:15.986886 16067 raft_consensus.cc:3037] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:09:15.989112 16201 raft_consensus.cc:2454] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82eb0c49468442e7b4b6f4029cf899c3 in term 1.
I20260620 03:09:15.989547 16267 leader_election.cc:302] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82eb0c49468442e7b4b6f4029cf899c3, c495ad6c2e444228b91cddf586a37693; no voters:
I20260620 03:09:15.989837 16405 raft_consensus.cc:2781] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:09:15.990044 16067 raft_consensus.cc:2454] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82eb0c49468442e7b4b6f4029cf899c3 in term 1.
I20260620 03:09:15.990371 16405 raft_consensus.cc:686] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [term 1 LEADER]: Becoming Leader. State: Replica: 82eb0c49468442e7b4b6f4029cf899c3, State: Running, Role: LEADER
I20260620 03:09:15.990885 16405 consensus_queue.cc:237] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } } peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } }
I20260620 03:09:15.995913 15923 catalog_manager.cc:5495] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82eb0c49468442e7b4b6f4029cf899c3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "82eb0c49468442e7b4b6f4029cf899c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } health_report { overall_health: UNKNOWN } } }
I20260620 03:09:16.040282 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:16.034947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56116 (local address 127.0.97.3:39969)
0620 03:09:16.035113 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:16.035117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:16.035136 (+ 19us) server_negotiation.cc:407] Connection header received
0620 03:09:16.035408 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:16.035413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:16.035492 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:16.035629 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:09:16.036017 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:16.036833 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:16.038375 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:16.038747 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:16.038840 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:16.039268 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:09:16.039822 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:09:16.039929 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:16.040029 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 03:09:16.040090 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 03:09:16.047996 16201 raft_consensus.cc:1261] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693 [term 1 FOLLOWER]: Refusing update from remote peer 82eb0c49468442e7b4b6f4029cf899c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:09:16.047991 16067 raft_consensus.cc:1261] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372 [term 1 FOLLOWER]: Refusing update from remote peer 82eb0c49468442e7b4b6f4029cf899c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:09:16.048975 16407 consensus_queue.cc:1035] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:09:16.049365 16405 consensus_queue.cc:1035] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:09:16.072307 16412 mvcc.cc:205] Tried to move back new op lower bound from 7298764619962253312 to 7298764619742859264. Current Snapshot: MvccSnapshot[applied={T|T < 7298764619962253312}]
I20260620 03:09:16.081931 15923 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:09:16.082145 15923 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:09:16.085485 15923 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: Sending DeleteTablet for 3 replicas of tablet 9f5d020bb94642c5a8ebbba3645791fc
I20260620 03:09:16.086612 16314 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f5d020bb94642c5a8ebbba3645791fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:09:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56114
I20260620 03:09:16.087319 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 15984
I20260620 03:09:16.091830 16181 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f5d020bb94642c5a8ebbba3645791fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:09:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54708
I20260620 03:09:16.093351 16047 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f5d020bb94642c5a8ebbba3645791fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:09:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41316
I20260620 03:09:16.110004 16424 tablet_replica.cc:330] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372: stopping tablet replica
I20260620 03:09:16.110550 16424 raft_consensus.cc:2229] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:09:16.112111 16424 raft_consensus.cc:2258] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:09:16.112824 16423 tablet_replica.cc:330] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693: stopping tablet replica
I20260620 03:09:16.113360 16423 raft_consensus.cc:2229] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:09:16.113922 16423 raft_consensus.cc:2258] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:09:16.116194 16423 ts_tablet_manager.cc:1905] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:09:16.116194 16424 ts_tablet_manager.cc:1905] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:09:16.122699 16421 tablet_replica.cc:330] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3: stopping tablet replica
I20260620 03:09:16.123486 16421 raft_consensus.cc:2229] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:09:16.124696 16421 raft_consensus.cc:2258] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:09:16.129101 16421 ts_tablet_manager.cc:1905] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 03:09:16 dist-test-slave-xv68 krb5kdc[15878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:09:16.135481 16424 ts_tablet_manager.cc:1918] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:09:16.135788 16424 log.cc:1196] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/9f5d020bb94642c5a8ebbba3645791fc
I20260620 03:09:16.135749 16423 ts_tablet_manager.cc:1918] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:09:16.136534 16423 log.cc:1196] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/9f5d020bb94642c5a8ebbba3645791fc
I20260620 03:09:16.138204 16423 ts_tablet_manager.cc:1939] T 9f5d020bb94642c5a8ebbba3645791fc P c495ad6c2e444228b91cddf586a37693: Deleting consensus metadata
I20260620 03:09:16.140072 16424 ts_tablet_manager.cc:1939] T 9f5d020bb94642c5a8ebbba3645791fc P deedccf28293476ebd2894c104fb6372: Deleting consensus metadata
I20260620 03:09:16.142067 15907 catalog_manager.cc:4841] TS deedccf28293476ebd2894c104fb6372 (127.0.97.1:40843): tablet 9f5d020bb94642c5a8ebbba3645791fc (table test-table [id=572811d54a664faeb6d3ffbf2a04bc0a]) successfully deleted
I20260620 03:09:16.142452 15907 catalog_manager.cc:4841] TS c495ad6c2e444228b91cddf586a37693 (127.0.97.2:33619): tablet 9f5d020bb94642c5a8ebbba3645791fc (table test-table [id=572811d54a664faeb6d3ffbf2a04bc0a]) successfully deleted
I20260620 03:09:16.144629 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:16.092476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41324 (local address 127.0.97.1:40843)
0620 03:09:16.092706 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:16.092714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:16.111327 (+ 18613us) server_negotiation.cc:407] Connection header received
0620 03:09:16.112624 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:16.112631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:16.112738 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:16.112833 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:16.117804 (+ 4971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:16.118888 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:16.123975 (+ 5087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:16.124654 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:16.124767 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:16.135854 (+ 11087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:16.135873 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:16.135892 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:16.135937 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:16.140041 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:16.141844 (+ 1803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:16.141856 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:16.141862 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:16.142027 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:16.142584 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:16.142589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:16.142592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:16.143066 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:16.143376 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:16.144008 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 03:09:16.144241 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 03:09:16.146080 16107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:09:16.148447 16421 ts_tablet_manager.cc:1918] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:09:16.148711 16421 log.cc:1196] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/9f5d020bb94642c5a8ebbba3645791fc
I20260620 03:09:16.149717 16421 ts_tablet_manager.cc:1939] T 9f5d020bb94642c5a8ebbba3645791fc P 82eb0c49468442e7b4b6f4029cf899c3: Deleting consensus metadata
I20260620 03:09:16.151690 15909 catalog_manager.cc:4841] TS 82eb0c49468442e7b4b6f4029cf899c3 (127.0.97.3:39969): tablet 9f5d020bb94642c5a8ebbba3645791fc (table test-table [id=572811d54a664faeb6d3ffbf2a04bc0a]) successfully deleted
I20260620 03:09:16.284449 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 15984
I20260620 03:09:16.302785 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 16118
Jun 20 03:09:16 dist-test-slave-xv68 krb5kdc[15878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:09:16.319872 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:16.303850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.0.97.2:33619)
0620 03:09:16.304138 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:16.304143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:16.304162 (+ 19us) server_negotiation.cc:407] Connection header received
0620 03:09:16.304219 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:16.304222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:16.304283 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:16.304377 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:16.306086 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:16.306786 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:16.308692 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:16.309080 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:16.309173 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:16.314000 (+ 4827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:16.314015 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:16.314030 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:16.314093 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:16.317083 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:16.317760 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:16.317766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:16.317769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:16.317899 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:16.318326 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:16.318337 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:16.318339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:16.318663 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:16.318972 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:16.319276 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 03:09:16.319531 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260620 03:09:16.320510 16241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:09:16.412238 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 16118
I20260620 03:09:16.435679 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 16251
Jun 20 03:09:16 dist-test-slave-xv68 krb5kdc[15878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:09:16.453075 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:16.436545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56132 (local address 127.0.97.3:39969)
0620 03:09:16.436741 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:16.436748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:16.436770 (+ 22us) server_negotiation.cc:407] Connection header received
0620 03:09:16.436854 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:16.436863 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:16.436943 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:16.437057 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:16.438451 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:16.439260 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:16.441315 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:16.441834 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:16.441933 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:16.447123 (+ 5190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:16.447135 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:16.447151 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:16.447192 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:16.450281 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:16.450968 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:16.450982 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:16.450984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:16.451137 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:16.451538 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:16.451544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:16.451547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:16.451900 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:16.452218 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:16.452499 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 03:09:16.452720 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 03:09:16.453752 16374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:09:16.576467 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 16251
I20260620 03:09:16.598359 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 15891
I20260620 03:09:16.599567 15953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:09:16.711773 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 15891
I20260620 03:09:36.733435 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38739
--webserver_interface=127.0.97.62
--webserver_port=44111
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:09:37.066193 16429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:09:37.066525 16429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:09:37.066701 16429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:09:37.073624 16429 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 03:09:37.073755 16429 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 03:09:37.073824 16429 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 03:09:37.073904 16429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:09:37.073979 16429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:09:37.074076 16429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:09:37.074159 16429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:09:37.074241 16429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:09:37.083834 16429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:09:37.085104 16429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:09:37.086395 16429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:09:37.109449 16435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:09:37.109537 16436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:09:37.109808 16438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:09:37.110828 16429 server_base.cc:1034] running on GCE node
I20260620 03:09:37.111919 16429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:09:37.113672 16429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:09:37.114902 16429 hybrid_clock.cc:648] HybridClock initialized: now 1781924977114840 us; error 34 us; skew 500 ppm
I20260620 03:09:37.115384 16429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:09:37 dist-test-slave-xv68 krb5kdc[15878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:09:37.119045 16429 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:09:37.120801 16429 webserver.cc:460] Webserver started at http://127.0.97.62:44111/ using document root <none> and password file <none>
I20260620 03:09:37.121613 16429 fs_manager.cc:359] Metadata directory not provided
I20260620 03:09:37.121765 16429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:09:37.127874 16429 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 03:09:37.130904 16445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:37.132486 16429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:09:37.132701 16429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "dc34559247524fc28ef40ba4961a7aa0"
format_stamp: "Formatted at 2026-06-20 03:09:13 on dist-test-slave-xv68"
server_key: "261bf68e77cbbbdc47b7f15303eef437"
server_key_iv: "c2d074a369ce2f207b8bdee860ff81f2"
server_key_version: "encryptionkey@0"
I20260620 03:09:37.133780 16429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:09:37.147161 16429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:09:37.151221 16429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:09:37.151546 16429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:09:37.170758 16429 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38739
I20260620 03:09:37.170809 16496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38739 every 8 connection(s)
I20260620 03:09:37.172359 16429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:09:37.177484 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 16429
I20260620 03:09:37.178326 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40843
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38739
--webserver_port=37357
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:09:37.179798 16497 sys_catalog.cc:263] Verifying existing consensus state
I20260620 03:09:37.183483 16497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: Bootstrap starting.
I20260620 03:09:37.203611 16497 log.cc:824] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: Log is configured to *not* fsync() on all Append() calls
I20260620 03:09:37.215528 16497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 03:09:37.216109 16497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: Bootstrap complete.
I20260620 03:09:37.226811 16497 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc34559247524fc28ef40ba4961a7aa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38739 } }
I20260620 03:09:37.227773 16497 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc34559247524fc28ef40ba4961a7aa0, State: Initialized, Role: FOLLOWER
I20260620 03:09:37.228458 16497 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc34559247524fc28ef40ba4961a7aa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38739 } }
I20260620 03:09:37.228767 16497 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:09:37.228895 16497 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:09:37.229100 16497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 03:09:37.233070 16497 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc34559247524fc28ef40ba4961a7aa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38739 } }
I20260620 03:09:37.233621 16497 leader_election.cc:302] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc34559247524fc28ef40ba4961a7aa0; no voters:
I20260620 03:09:37.234294 16497 leader_election.cc:288] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 03:09:37.234437 16501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 03:09:37.235687 16501 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 2 LEADER]: Becoming Leader. State: Replica: dc34559247524fc28ef40ba4961a7aa0, State: Running, Role: LEADER
I20260620 03:09:37.236327 16501 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc34559247524fc28ef40ba4961a7aa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38739 } }
I20260620 03:09:37.236640 16497 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:09:37.239428 16502 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dc34559247524fc28ef40ba4961a7aa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc34559247524fc28ef40ba4961a7aa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38739 } } }
I20260620 03:09:37.240283 16502 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [sys.catalog]: This master's current role is: LEADER
I20260620 03:09:37.239423 16503 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc34559247524fc28ef40ba4961a7aa0. Latest consensus state: current_term: 2 leader_uuid: "dc34559247524fc28ef40ba4961a7aa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc34559247524fc28ef40ba4961a7aa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38739 } } }
I20260620 03:09:37.240963 16503 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [sys.catalog]: This master's current role is: LEADER
I20260620 03:09:37.250650 16507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:09:37.259167 16507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:09:37.260624 16507 catalog_manager.cc:1259] Loaded cluster ID: 56bb5e45351a49d884f122283afcd123
I20260620 03:09:37.260793 16507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:09:37.265069 16507 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:09:37.266992 16507 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: Loaded TSK: 0
I20260620 03:09:37.301416 16507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: Generated new TSK 1
I20260620 03:09:37.301846 16507 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 03:09:37.539079 16499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:09:37.539433 16499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:09:37.539613 16499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:09:37.546923 16499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:09:37.547118 16499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:09:37.547355 16499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:09:37.557619 16499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37357
--tserver_master_addrs=127.0.97.62:38739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:09:37.558893 16499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:09:37.560181 16499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:09:37.574757 16524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:09:37.574790 16525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:09:37.574967 16527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:09:37.576836 16499 server_base.cc:1034] running on GCE node
I20260620 03:09:37.577490 16499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:09:37.578509 16499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:09:37.579757 16499 hybrid_clock.cc:648] HybridClock initialized: now 1781924977579695 us; error 50 us; skew 500 ppm
I20260620 03:09:37.580091 16499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:09:37 dist-test-slave-xv68 krb5kdc[15878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:09:37.583519 16499 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:09:37.585089 16499 webserver.cc:460] Webserver started at http://127.0.97.1:37357/ using document root <none> and password file <none>
I20260620 03:09:37.585884 16499 fs_manager.cc:359] Metadata directory not provided
I20260620 03:09:37.586030 16499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:09:37.591756 16499 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 03:09:37.594702 16534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:37.596225 16499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:09:37.596433 16499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "deedccf28293476ebd2894c104fb6372"
format_stamp: "Formatted at 2026-06-20 03:09:14 on dist-test-slave-xv68"
server_key: "ba2a3f9b6493c6f14d950ab98e74cd8e"
server_key_iv: "f8829fb2d1300b2302915e106dda1048"
server_key_version: "encryptionkey@0"
I20260620 03:09:37.597450 16499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:09:37.615077 16499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:09:37.619552 16499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:09:37.619841 16499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:09:37.620981 16499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:09:37.622463 16499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:09:37.622586 16499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:37.622735 16499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:09:37.622825 16499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:37.659207 16499 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40843
I20260620 03:09:37.659355 16646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40843 every 8 connection(s)
I20260620 03:09:37.660670 16499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:09:37.662971 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 16499
I20260620 03:09:37.663837 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33619
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38739
--webserver_port=40109
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:09:37 dist-test-slave-xv68 krb5kdc[15878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:09:37.683669 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:37.663574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38853 (local address 127.0.97.62:38739)
0620 03:09:37.664246 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:37.664261 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:37.664903 (+ 642us) server_negotiation.cc:407] Connection header received
0620 03:09:37.666957 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:37.666970 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:37.667278 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:37.667690 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:37.669172 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:37.670396 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:37.672025 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:37.672656 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:37.672855 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:37.676236 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:37.676244 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:37.676257 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:37.676292 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:37.679200 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:37.679981 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:37.679988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:37.679993 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:37.680150 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:37.680598 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:37.680607 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:37.680609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:37.681063 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:37.681279 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:37.682046 (+ 767us) server_negotiation.cc:299] Negotiation successful
0620 03:09:37.682414 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":307,"threads_started":1}
I20260620 03:09:37.683681 16650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:37.663915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:38853)
0620 03:09:37.664730 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:37.664765 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 03:09:37.666428 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:09:37.667841 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:09:37.667852 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:09:37.668451 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:09:37.668962 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:37.668979 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:37.670569 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:37.670576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:37.671872 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:37.671884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:37.672893 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:37.672900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:37.673166 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:37.673837 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:09:37.673859 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:09:37.675957 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:09:37.679378 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:09:37.679388 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:09:37.679402 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:09:37.679830 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:09:37.680276 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:09:37.680282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:09:37.680286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:09:37.680474 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:09:37.681211 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:09:37.681219 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:09:37.681536 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 03:09:37.681827 (+ 291us) client_negotiation.cc:240] Negotiation successful
0620 03:09:37.682356 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":378,"threads_started":1}
I20260620 03:09:37.686995 16647 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38739
I20260620 03:09:37.687343 16647 heartbeater.cc:464] Registering TS with master...
I20260620 03:09:37.688081 16647 heartbeater.cc:511] Master 127.0.97.62:38739 requested a full tablet report, sending...
I20260620 03:09:37.690522 16462 ts_manager.cc:194] Registered new tserver with Master: deedccf28293476ebd2894c104fb6372 (127.0.97.1:40843)
I20260620 03:09:37.692705 16462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38853
W20260620 03:09:37.999420 16651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:09:37.999698 16651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:09:37.999792 16651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:09:38.007668 16651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:09:38.007777 16651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:09:38.007889 16651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:09:38.017902 16651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40109
--tserver_master_addrs=127.0.97.62:38739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:09:38.019080 16651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:09:38.020401 16651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:09:38.034111 16656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:09:38.034153 16659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:09:38.034210 16657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:09:38.040218 16651 server_base.cc:1034] running on GCE node
I20260620 03:09:38.040915 16651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:09:38.041975 16651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:09:38.043208 16651 hybrid_clock.cc:648] HybridClock initialized: now 1781924978043163 us; error 56 us; skew 500 ppm
I20260620 03:09:38.043553 16651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:09:38 dist-test-slave-xv68 krb5kdc[15878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:09:38.046602 16651 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:09:38.048192 16651 webserver.cc:460] Webserver started at http://127.0.97.2:40109/ using document root <none> and password file <none>
I20260620 03:09:38.048982 16651 fs_manager.cc:359] Metadata directory not provided
I20260620 03:09:38.049135 16651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:09:38.054441 16651 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 03:09:38.057178 16666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:38.058418 16651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:09:38.058599 16651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "c495ad6c2e444228b91cddf586a37693"
format_stamp: "Formatted at 2026-06-20 03:09:15 on dist-test-slave-xv68"
server_key: "5124765e3bfbda5157b26fc791a49903"
server_key_iv: "2b477e463df3aec3fb93c44376025421"
server_key_version: "encryptionkey@0"
I20260620 03:09:38.059620 16651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:09:38.070950 16651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:09:38.074635 16651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:09:38.074927 16651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:09:38.075877 16651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:09:38.077255 16651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:09:38.077371 16651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:38.077503 16651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:09:38.077591 16651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:38.110960 16651 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33619
I20260620 03:09:38.111047 16778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33619 every 8 connection(s)
I20260620 03:09:38.112416 16651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:09:38.117404 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 16651
I20260620 03:09:38.118024 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39969
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38739
--webserver_port=43827
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:09:38 dist-test-slave-xv68 krb5kdc[15878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:09:38.135186 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.114843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53155 (local address 127.0.97.62:38739)
0620 03:09:38.115048 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:38.115054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:38.116242 (+ 1188us) server_negotiation.cc:407] Connection header received
0620 03:09:38.118068 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:38.118074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:38.118173 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:38.118270 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:38.120395 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.121070 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.123652 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.124326 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.124448 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.128445 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:38.128454 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:38.128462 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:38.128510 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:38.132021 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:38.132709 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:38.132716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:38.132722 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:38.132863 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:38.133248 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:38.133255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:38.133261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:38.133694 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:38.133844 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:38.134473 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 03:09:38.134720 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 03:09:38.135828 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.115415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:53155)
0620 03:09:38.116084 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:38.116128 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 03:09:38.117810 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:09:38.118531 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:09:38.118546 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:09:38.119413 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:09:38.120204 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:38.120230 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.121228 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.121236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:38.123489 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:38.123506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.124570 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.124579 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:38.124911 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.125708 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:09:38.125742 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:09:38.128176 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:09:38.132149 (+ 3973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:09:38.132156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:09:38.132168 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:09:38.132572 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:09:38.132966 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:09:38.132969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:09:38.132971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:09:38.133134 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:09:38.133814 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:09:38.133820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:09:38.134107 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 03:09:38.134323 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 03:09:38.134694 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":266,"threads_started":1}
I20260620 03:09:38.137336 16779 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38739
I20260620 03:09:38.137712 16779 heartbeater.cc:464] Registering TS with master...
I20260620 03:09:38.138525 16779 heartbeater.cc:511] Master 127.0.97.62:38739 requested a full tablet report, sending...
I20260620 03:09:38.140012 16462 ts_manager.cc:194] Registered new tserver with Master: c495ad6c2e444228b91cddf586a37693 (127.0.97.2:33619)
I20260620 03:09:38.140812 16462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53155
W20260620 03:09:38.450284 16782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:09:38.450543 16782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:09:38.450639 16782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:09:38.457835 16782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:09:38.457938 16782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:09:38.458050 16782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:09:38.467535 16782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43827
--tserver_master_addrs=127.0.97.62:38739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:09:38.468681 16782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:09:38.469913 16782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:09:38.482729 16787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:09:38.482772 16788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:09:38.482930 16790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:09:38.483769 16782 server_base.cc:1034] running on GCE node
I20260620 03:09:38.484391 16782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:09:38.485261 16782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:09:38.486454 16782 hybrid_clock.cc:648] HybridClock initialized: now 1781924978486411 us; error 43 us; skew 500 ppm
I20260620 03:09:38.486742 16782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:09:38 dist-test-slave-xv68 krb5kdc[15878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:09:38.489670 16782 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:09:38.491354 16782 webserver.cc:460] Webserver started at http://127.0.97.3:43827/ using document root <none> and password file <none>
I20260620 03:09:38.492069 16782 fs_manager.cc:359] Metadata directory not provided
I20260620 03:09:38.492233 16782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:09:38.497603 16782 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 03:09:38.500598 16797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:38.501998 16782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:09:38.502167 16782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "82eb0c49468442e7b4b6f4029cf899c3"
format_stamp: "Formatted at 2026-06-20 03:09:15 on dist-test-slave-xv68"
server_key: "fb6af908d8a3d3f49a8d1710936d65f5"
server_key_iv: "d02459d083c75280e7378b4ee36405b2"
server_key_version: "encryptionkey@0"
I20260620 03:09:38.503142 16782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:09:38.518594 16782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:09:38.522717 16782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:09:38.523000 16782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:09:38.524115 16782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:09:38.525568 16782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:09:38.525669 16782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:38.525826 16782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:09:38.525918 16782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:38.560072 16782 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39969
I20260620 03:09:38.560153 16909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39969 every 8 connection(s)
I20260620 03:09:38.561465 16782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:09:38.565960 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 16782
Jun 20 03:09:38 dist-test-slave-xv68 krb5kdc[15878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:09:38.584774 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.567040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.0.97.62:38739)
0620 03:09:38.568440 (+ 1400us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:38.568444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:38.568457 (+ 13us) server_negotiation.cc:407] Connection header received
0620 03:09:38.568499 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:38.568502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:38.568549 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:38.568631 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:38.570674 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.571731 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.573766 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.574649 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.574877 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.578479 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:38.578486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:38.578490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:38.578534 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:38.581453 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:38.582301 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:38.582304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:38.582307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:38.582425 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:38.582860 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:38.582865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:38.582868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:38.583286 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:38.583578 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:38.584067 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 03:09:38.584316 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":222,"server-negotiator.queue_time_us":1312,"thread_start_us":1186,"threads_started":1}
I20260620 03:09:38.586385 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.564071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59929 (local address 127.0.97.62:38739)
0620 03:09:38.564293 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:38.564300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:38.565399 (+ 1099us) server_negotiation.cc:407] Connection header received
0620 03:09:38.568019 (+ 2620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:38.568023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:38.568086 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:38.568199 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:38.570320 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.571472 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.573766 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.574649 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.574876 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.579245 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:38.579253 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:38.579259 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:38.579301 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:38.582621 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:38.583677 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:38.583682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:38.583685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:38.583851 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:38.584385 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:38.584390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:38.584394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:38.584768 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:38.584914 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:38.585809 (+ 895us) server_negotiation.cc:299] Negotiation successful
0620 03:09:38.586020 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":75}
I20260620 03:09:38.587461 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.564524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:59929)
0620 03:09:38.565213 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:38.565256 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 03:09:38.567671 (+ 2415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:09:38.568451 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:09:38.568460 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:09:38.569232 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:09:38.570105 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:38.570136 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.571618 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.571622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:38.573580 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:38.573593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.574903 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.574912 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:38.575195 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.576015 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:09:38.576051 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:09:38.578977 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:09:38.582835 (+ 3858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:09:38.582846 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:09:38.582865 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:09:38.583548 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:09:38.583983 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:09:38.583988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:09:38.583991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:09:38.584244 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:09:38.584933 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:09:38.584945 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:09:38.585370 (+ 425us) client_negotiation.cc:769] Sending connection context
0620 03:09:38.585673 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 03:09:38.586120 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":275,"threads_started":1}
I20260620 03:09:38.588778 16910 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38739
I20260620 03:09:38.589040 16910 heartbeater.cc:464] Registering TS with master...
I20260620 03:09:38.589687 16910 heartbeater.cc:511] Master 127.0.97.62:38739 requested a full tablet report, sending...
I20260620 03:09:38.590984 16462 ts_manager.cc:194] Registered new tserver with Master: 82eb0c49468442e7b4b6f4029cf899c3 (127.0.97.3:39969)
I20260620 03:09:38.591850 16462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59929
I20260620 03:09:38.600042 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:09:38.606679 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.600998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.0.97.62:38739)
0620 03:09:38.601163 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:38.601167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:38.601613 (+ 446us) server_negotiation.cc:407] Connection header received
0620 03:09:38.601855 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:38.601858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:38.601994 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:38.602129 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:09:38.602559 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.603265 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.604943 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.605346 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.605451 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.605863 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:09:38.606054 (+ 191us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 03:09:38.606399 (+ 345us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55546: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 03:09:38.606798 16649 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55546: token expired
W20260620 03:09:38.607506 16384 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:38739)
I20260620 03:09:38.620369 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.608063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55560 (local address 127.0.97.62:38739)
0620 03:09:38.608249 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:38.608255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:38.608272 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:09:38.608426 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:38.608431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:38.608500 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:38.608598 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:38.610179 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.610897 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.612287 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.612696 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.612827 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.615330 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:38.615336 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:38.615341 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:38.615384 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:38.617749 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:38.618415 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:38.618420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:38.618424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:38.618549 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:38.618913 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:38.618919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:38.618922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:38.619234 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:38.619507 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:38.619837 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 03:09:38.620055 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 03:09:38.623970 16462 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:09:38.631372 16462 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:09:38.672086 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.643597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.0.97.3:39969)
0620 03:09:38.662959 (+ 19362us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:38.662972 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:38.663004 (+ 32us) server_negotiation.cc:407] Connection header received
0620 03:09:38.663104 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:38.663111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:38.663352 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:38.663632 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:09:38.664623 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.667249 (+ 2626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.669387 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.670802 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.670928 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.671165 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:38.671562 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 03:09:38.671734 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19143,"thread_start_us":23229,"threads_started":1}
I20260620 03:09:38.672127 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.643324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39969 (local address 127.0.0.1:33398)
0620 03:09:38.657873 (+ 14549us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:38.657888 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:09:38.657977 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:09:38.663910 (+ 5933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:09:38.663917 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:09:38.663947 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:09:38.664408 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:38.664420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.667458 (+ 3038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.667464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:38.669157 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:38.669171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.671050 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.671058 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:38.671277 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.671358 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 03:09:38.671493 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 03:09:38.671674 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14457,"thread_start_us":13565,"threads_started":1}
I20260620 03:09:38.674089 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.643023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40843 (local address 127.0.0.1:46346)
0620 03:09:38.656327 (+ 13304us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:38.656397 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 03:09:38.656650 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:09:38.665660 (+ 9010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:09:38.665667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:09:38.665700 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:09:38.666185 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:38.666200 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.668935 (+ 2735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.668941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:38.671078 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:38.671092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.673297 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.673307 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:38.673483 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.673520 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 03:09:38.673617 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 03:09:38.673763 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12937,"thread_start_us":14999,"threads_started":1}
I20260620 03:09:38.674875 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.643230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46346 (local address 127.0.97.1:40843)
0620 03:09:38.661390 (+ 18160us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:38.661400 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:38.661428 (+ 28us) server_negotiation.cc:407] Connection header received
0620 03:09:38.661531 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:38.661541 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:38.661828 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:38.662030 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:09:38.666431 (+ 4401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.668762 (+ 2331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.671317 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.672771 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.672897 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.673100 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:38.674380 (+ 1280us) server_negotiation.cc:299] Negotiation successful
0620 03:09:38.674560 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18004,"thread_start_us":19165,"threads_started":1}
I20260620 03:09:38.676270 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.658376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33619 (local address 127.0.0.1:41668)
0620 03:09:38.665146 (+ 6770us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:38.665170 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 03:09:38.665311 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:09:38.669778 (+ 4467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:09:38.669782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:09:38.669807 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:09:38.670185 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:38.670194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.671799 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.671806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:38.673796 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:38.673810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.675495 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.675502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:38.675686 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.675715 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 03:09:38.675806 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 03:09:38.675940 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6591,"thread_start_us":7030,"threads_started":1}
I20260620 03:09:38.676467 16845 tablet_service.cc:1467] Processing CreateTablet for tablet fb48893d75d84bca8c6f21dc3ce660a7 (DEFAULT_TABLE table=test-table [id=f71d60b6117e47f49153dc11171ba512]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:09:38.677002 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.658646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.0.97.2:33619)
0620 03:09:38.669331 (+ 10685us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:38.669336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:38.669358 (+ 22us) server_negotiation.cc:407] Connection header received
0620 03:09:38.669430 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:38.669435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:38.669584 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:38.669791 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:09:38.670360 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.671642 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.673997 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.675241 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.675435 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.675631 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:38.676498 (+ 867us) server_negotiation.cc:299] Negotiation successful
0620 03:09:38.676707 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10530,"thread_start_us":10437,"threads_started":1}
I20260620 03:09:38.677819 16845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb48893d75d84bca8c6f21dc3ce660a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:09:38.678263 16582 tablet_service.cc:1467] Processing CreateTablet for tablet fb48893d75d84bca8c6f21dc3ce660a7 (DEFAULT_TABLE table=test-table [id=f71d60b6117e47f49153dc11171ba512]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:09:38.679446 16582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb48893d75d84bca8c6f21dc3ce660a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:09:38.684638 16714 tablet_service.cc:1467] Processing CreateTablet for tablet fb48893d75d84bca8c6f21dc3ce660a7 (DEFAULT_TABLE table=test-table [id=f71d60b6117e47f49153dc11171ba512]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:09:38.685737 16714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb48893d75d84bca8c6f21dc3ce660a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:09:38.695982 16647 heartbeater.cc:503] Master 127.0.97.62:38739 was elected leader, sending a full tablet report...
I20260620 03:09:38.700337 16926 tablet_bootstrap.cc:492] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372: Bootstrap starting.
I20260620 03:09:38.700331 16928 tablet_bootstrap.cc:492] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693: Bootstrap starting.
I20260620 03:09:38.704068 16927 tablet_bootstrap.cc:492] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3: Bootstrap starting.
I20260620 03:09:38.704670 16926 tablet_bootstrap.cc:654] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372: Neither blocks nor log segments found. Creating new log.
I20260620 03:09:38.704905 16928 tablet_bootstrap.cc:654] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693: Neither blocks nor log segments found. Creating new log.
I20260620 03:09:38.706040 16928 log.cc:824] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693: Log is configured to *not* fsync() on all Append() calls
I20260620 03:09:38.706043 16926 log.cc:824] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372: Log is configured to *not* fsync() on all Append() calls
I20260620 03:09:38.707404 16927 tablet_bootstrap.cc:654] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3: Neither blocks nor log segments found. Creating new log.
I20260620 03:09:38.708635 16927 log.cc:824] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3: Log is configured to *not* fsync() on all Append() calls
I20260620 03:09:38.709741 16926 tablet_bootstrap.cc:492] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372: No bootstrap required, opened a new log
I20260620 03:09:38.710063 16926 ts_tablet_manager.cc:1397] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 03:09:38.710153 16928 tablet_bootstrap.cc:492] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693: No bootstrap required, opened a new log
I20260620 03:09:38.710513 16928 ts_tablet_manager.cc:1397] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260620 03:09:38.717451 16926 raft_consensus.cc:348] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } }
I20260620 03:09:38.717864 16926 raft_consensus.cc:374] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:09:38.717995 16926 raft_consensus.cc:729] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deedccf28293476ebd2894c104fb6372, State: Initialized, Role: FOLLOWER
I20260620 03:09:38.718163 16927 tablet_bootstrap.cc:492] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3: No bootstrap required, opened a new log
I20260620 03:09:38.718441 16927 ts_tablet_manager.cc:1397] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260620 03:09:38.718695 16926 consensus_queue.cc:260] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } }
I20260620 03:09:38.721308 16926 ts_tablet_manager.cc:1428] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260620 03:09:38.721971 16928 raft_consensus.cc:348] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } }
I20260620 03:09:38.722433 16928 raft_consensus.cc:374] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:09:38.722532 16928 raft_consensus.cc:729] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c495ad6c2e444228b91cddf586a37693, State: Initialized, Role: FOLLOWER
I20260620 03:09:38.723203 16928 consensus_queue.cc:260] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } }
I20260620 03:09:38.724715 16779 heartbeater.cc:503] Master 127.0.97.62:38739 was elected leader, sending a full tablet report...
I20260620 03:09:38.724972 16928 ts_tablet_manager.cc:1428] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693: Time spent starting tablet: real 0.014s user 0.014s sys 0.002s
I20260620 03:09:38.728353 16927 raft_consensus.cc:348] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } }
I20260620 03:09:38.728725 16927 raft_consensus.cc:374] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:09:38.728824 16927 raft_consensus.cc:729] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82eb0c49468442e7b4b6f4029cf899c3, State: Initialized, Role: FOLLOWER
I20260620 03:09:38.729417 16927 consensus_queue.cc:260] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } }
I20260620 03:09:38.730322 16910 heartbeater.cc:503] Master 127.0.97.62:38739 was elected leader, sending a full tablet report...
I20260620 03:09:38.730784 16927 ts_tablet_manager.cc:1428] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
W20260620 03:09:38.813181 16911 tablet.cc:2369] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:09:38.825169 16934 raft_consensus.cc:482] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:09:38.825500 16934 raft_consensus.cc:504] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } }
I20260620 03:09:38.827253 16934 leader_election.cc:288] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers deedccf28293476ebd2894c104fb6372 (127.0.97.1:40843), c495ad6c2e444228b91cddf586a37693 (127.0.97.2:33619)
I20260620 03:09:38.833673 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.827751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40843 (local address 127.0.97.3:55993)
0620 03:09:38.827974 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:38.827990 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:09:38.828104 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:09:38.828698 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:09:38.828702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:09:38.828731 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:09:38.828924 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:38.828933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.830164 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.830167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:38.831964 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:38.831978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.833150 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.833154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:38.833279 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.833303 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 03:09:38.833373 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 03:09:38.833465 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260620 03:09:38.833810 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.827745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55993 (local address 127.0.97.1:40843)
0620 03:09:38.827977 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:38.827983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:38.828051 (+ 68us) server_negotiation.cc:407] Connection header received
0620 03:09:38.828284 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:38.828292 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:38.828378 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:38.828483 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:09:38.829065 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.830036 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.832107 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.833010 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.833148 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.833247 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:38.833417 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 03:09:38.833549 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 03:09:38.834833 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.828492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58305 (local address 127.0.97.2:33619)
0620 03:09:38.828720 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:38.828728 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:38.829062 (+ 334us) server_negotiation.cc:407] Connection header received
0620 03:09:38.829326 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:38.829331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:38.829411 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:38.829504 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:09:38.830017 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.830738 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.832766 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.833975 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.834107 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.834210 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:38.834562 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 03:09:38.834654 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 03:09:38.834998 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.828232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33619 (local address 127.0.97.3:58305)
0620 03:09:38.828926 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 03:09:38.828977 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 03:09:38.829160 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:09:38.829549 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:09:38.829553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:09:38.829576 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:09:38.829864 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:38.829870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.830882 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.830888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:38.832617 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:09:38.832628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.834193 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.834197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:09:38.834390 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.834429 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 03:09:38.834529 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 03:09:38.834667 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":371,"threads_started":1}
I20260620 03:09:38.834802 16602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb48893d75d84bca8c6f21dc3ce660a7" candidate_uuid: "82eb0c49468442e7b4b6f4029cf899c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deedccf28293476ebd2894c104fb6372" is_pre_election: true
I20260620 03:09:38.835465 16602 raft_consensus.cc:2454] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82eb0c49468442e7b4b6f4029cf899c3 in term 0.
I20260620 03:09:38.836079 16734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb48893d75d84bca8c6f21dc3ce660a7" candidate_uuid: "82eb0c49468442e7b4b6f4029cf899c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c495ad6c2e444228b91cddf586a37693" is_pre_election: true
I20260620 03:09:38.836305 16798 leader_election.cc:302] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82eb0c49468442e7b4b6f4029cf899c3, deedccf28293476ebd2894c104fb6372; no voters:
I20260620 03:09:38.836642 16734 raft_consensus.cc:2454] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82eb0c49468442e7b4b6f4029cf899c3 in term 0.
I20260620 03:09:38.836761 16934 raft_consensus.cc:2781] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:09:38.836887 16934 raft_consensus.cc:482] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:09:38.837019 16934 raft_consensus.cc:3037] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:09:38.839375 16934 raft_consensus.cc:504] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } }
I20260620 03:09:38.839910 16934 leader_election.cc:288] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [CANDIDATE]: Term 1 election: Requested vote from peers deedccf28293476ebd2894c104fb6372 (127.0.97.1:40843), c495ad6c2e444228b91cddf586a37693 (127.0.97.2:33619)
I20260620 03:09:38.840329 16602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb48893d75d84bca8c6f21dc3ce660a7" candidate_uuid: "82eb0c49468442e7b4b6f4029cf899c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deedccf28293476ebd2894c104fb6372"
I20260620 03:09:38.840381 16734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb48893d75d84bca8c6f21dc3ce660a7" candidate_uuid: "82eb0c49468442e7b4b6f4029cf899c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c495ad6c2e444228b91cddf586a37693"
I20260620 03:09:38.840520 16602 raft_consensus.cc:3037] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:09:38.840605 16734 raft_consensus.cc:3037] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:09:38.842739 16602 raft_consensus.cc:2454] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82eb0c49468442e7b4b6f4029cf899c3 in term 1.
I20260620 03:09:38.842806 16734 raft_consensus.cc:2454] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82eb0c49468442e7b4b6f4029cf899c3 in term 1.
I20260620 03:09:38.843152 16798 leader_election.cc:302] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82eb0c49468442e7b4b6f4029cf899c3, deedccf28293476ebd2894c104fb6372; no voters:
I20260620 03:09:38.843457 16934 raft_consensus.cc:2781] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:09:38.843864 16934 raft_consensus.cc:686] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [term 1 LEADER]: Becoming Leader. State: Replica: 82eb0c49468442e7b4b6f4029cf899c3, State: Running, Role: LEADER
I20260620 03:09:38.844312 16934 consensus_queue.cc:237] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } }
I20260620 03:09:38.848464 16462 catalog_manager.cc:5495] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82eb0c49468442e7b4b6f4029cf899c3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "82eb0c49468442e7b4b6f4029cf899c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82eb0c49468442e7b4b6f4029cf899c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } health_report { overall_health: UNKNOWN } } }
W20260620 03:09:38.864363 16780 tablet.cc:2369] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:09:38.893824 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.888667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.0.97.3:39969)
0620 03:09:38.888859 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:38.888863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:38.888879 (+ 16us) server_negotiation.cc:407] Connection header received
0620 03:09:38.888964 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:38.888968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:38.889039 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:38.889155 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:09:38.889501 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.890308 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.891812 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.892210 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.892311 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.892742 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:09:38.893216 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:09:38.893323 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:38.893442 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 03:09:38.893506 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 03:09:38.901892 16734 raft_consensus.cc:1261] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693 [term 1 FOLLOWER]: Refusing update from remote peer 82eb0c49468442e7b4b6f4029cf899c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:09:38.901926 16602 raft_consensus.cc:1261] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372 [term 1 FOLLOWER]: Refusing update from remote peer 82eb0c49468442e7b4b6f4029cf899c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:09:38.902884 16934 consensus_queue.cc:1035] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c495ad6c2e444228b91cddf586a37693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:09:38.903191 16936 consensus_queue.cc:1035] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "deedccf28293476ebd2894c104fb6372" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 03:09:38.912957 16648 tablet.cc:2369] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:09:38.915438 16938 mvcc.cc:205] Tried to move back new op lower bound from 7298764713571999744 to 7298764713349611520. Current Snapshot: MvccSnapshot[applied={T|T < 7298764713571999744}]
I20260620 03:09:38.918118 16940 mvcc.cc:205] Tried to move back new op lower bound from 7298764713571999744 to 7298764713349611520. Current Snapshot: MvccSnapshot[applied={T|T < 7298764713571999744}]
I20260620 03:09:38.929889 16461 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:09:38.930233 16461 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:09:38.934175 16461 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: Sending DeleteTablet for 3 replicas of tablet fb48893d75d84bca8c6f21dc3ce660a7
I20260620 03:09:38.935482 16714 tablet_service.cc:1514] Processing DeleteTablet for tablet fb48893d75d84bca8c6f21dc3ce660a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:09:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41668
I20260620 03:09:38.935490 16845 tablet_service.cc:1514] Processing DeleteTablet for tablet fb48893d75d84bca8c6f21dc3ce660a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:09:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33398
I20260620 03:09:38.935573 16582 tablet_service.cc:1514] Processing DeleteTablet for tablet fb48893d75d84bca8c6f21dc3ce660a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:09:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46346
I20260620 03:09:38.936892 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 16499
I20260620 03:09:38.951143 16950 tablet_replica.cc:330] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372: stopping tablet replica
I20260620 03:09:38.951841 16950 raft_consensus.cc:2229] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:09:38.952522 16950 raft_consensus.cc:2258] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:09:38.953569 16948 tablet_replica.cc:330] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3: stopping tablet replica
I20260620 03:09:38.954078 16948 raft_consensus.cc:2229] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:09:38.954844 16948 raft_consensus.cc:2258] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:09:38.955519 16949 tablet_replica.cc:330] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693: stopping tablet replica
I20260620 03:09:38.956171 16949 raft_consensus.cc:2229] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:09:38.956743 16949 raft_consensus.cc:2258] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:09:38.958276 16948 ts_tablet_manager.cc:1905] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:09:38.958711 16949 ts_tablet_manager.cc:1905] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:09:38.960480 16950 ts_tablet_manager.cc:1905] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:09:38.965593 16949 ts_tablet_manager.cc:1918] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:09:38.965811 16949 log.cc:1196] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/fb48893d75d84bca8c6f21dc3ce660a7
I20260620 03:09:38.966403 16949 ts_tablet_manager.cc:1939] T fb48893d75d84bca8c6f21dc3ce660a7 P c495ad6c2e444228b91cddf586a37693: Deleting consensus metadata
I20260620 03:09:38.966604 16948 ts_tablet_manager.cc:1918] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:09:38.967689 16950 ts_tablet_manager.cc:1918] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:09:38.967911 16950 log.cc:1196] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/fb48893d75d84bca8c6f21dc3ce660a7
I20260620 03:09:38.968344 16446 catalog_manager.cc:4841] TS c495ad6c2e444228b91cddf586a37693 (127.0.97.2:33619): tablet fb48893d75d84bca8c6f21dc3ce660a7 (table test-table [id=f71d60b6117e47f49153dc11171ba512]) successfully deleted
I20260620 03:09:38.968547 16950 ts_tablet_manager.cc:1939] T fb48893d75d84bca8c6f21dc3ce660a7 P deedccf28293476ebd2894c104fb6372: Deleting consensus metadata
I20260620 03:09:38.968851 16948 log.cc:1196] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/fb48893d75d84bca8c6f21dc3ce660a7
I20260620 03:09:38.969434 16948 ts_tablet_manager.cc:1939] T fb48893d75d84bca8c6f21dc3ce660a7 P 82eb0c49468442e7b4b6f4029cf899c3: Deleting consensus metadata
I20260620 03:09:38.970564 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:38.939001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.0.97.1:40843)
0620 03:09:38.939228 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:38.939237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:38.939261 (+ 24us) server_negotiation.cc:407] Connection header received
0620 03:09:38.939353 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:38.939361 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:38.939492 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:38.939626 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:38.942276 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.943912 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.946626 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:38.947631 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:38.947784 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:38.955610 (+ 7826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:38.955626 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:38.955643 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:38.955684 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:38.959530 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:38.960423 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:38.960431 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:38.960434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:38.966901 (+ 6467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:38.967458 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:38.967466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:38.967472 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:38.968000 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:38.968318 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:38.969864 (+ 1546us) server_negotiation.cc:299] Negotiation successful
0620 03:09:38.970127 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 03:09:38.970579 16446 catalog_manager.cc:4841] TS deedccf28293476ebd2894c104fb6372 (127.0.97.1:40843): tablet fb48893d75d84bca8c6f21dc3ce660a7 (table test-table [id=f71d60b6117e47f49153dc11171ba512]) successfully deleted
I20260620 03:09:38.971103 16448 catalog_manager.cc:4841] TS 82eb0c49468442e7b4b6f4029cf899c3 (127.0.97.3:39969): tablet fb48893d75d84bca8c6f21dc3ce660a7 (table test-table [id=f71d60b6117e47f49153dc11171ba512]) successfully deleted
I20260620 03:09:38.971415 16642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:09:39.063108 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 16499
I20260620 03:09:39.080533 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 16651
I20260620 03:09:39.095989 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:39.081307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41680 (local address 127.0.97.2:33619)
0620 03:09:39.081487 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:39.081491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:39.081525 (+ 34us) server_negotiation.cc:407] Connection header received
0620 03:09:39.081765 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:39.081769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:39.081838 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:39.081930 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:39.083699 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:39.084433 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:39.086258 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:39.086676 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:39.086769 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:39.090321 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:39.090335 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:39.090351 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:39.090383 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:39.093268 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:39.094107 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:39.094115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:39.094117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:39.094235 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:39.094611 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:39.094615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:39.094619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:39.094971 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:39.095209 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:39.095532 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 03:09:39.095692 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 03:09:39.096652 16774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:09:39.191479 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 16651
I20260620 03:09:39.209266 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 16782
I20260620 03:09:39.226169 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:09:39.210065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.0.97.3:39969)
0620 03:09:39.210259 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 03:09:39.210264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:09:39.210332 (+ 68us) server_negotiation.cc:407] Connection header received
0620 03:09:39.210567 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:09:39.210572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:09:39.210647 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:09:39.210743 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:09:39.212666 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:39.213542 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:39.215552 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:09:39.216024 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:09:39.216150 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:09:39.219494 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:09:39.219506 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:09:39.219522 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:09:39.219561 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:09:39.222748 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:39.223798 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:39.223805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:39.223807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:39.223964 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:09:39.224473 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:09:39.224477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:09:39.224479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:09:39.224872 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:09:39.225117 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 03:09:39.225529 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 03:09:39.225725 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 03:09:39.226933 16905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:09:39.350028 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 16782
I20260620 03:09:39.369612 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 16429
I20260620 03:09:39.370746 16492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:09:39.487699 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 16429
I20260620 03:09:59.509269 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38739
--webserver_interface=127.0.97.62
--webserver_port=44111
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:09:59.837433 16955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:09:59.837800 16955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:09:59.837970 16955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:09:59.845500 16955 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 03:09:59.845650 16955 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 03:09:59.845745 16955 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 03:09:59.845845 16955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:09:59.845948 16955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:09:59.846072 16955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:09:59.846165 16955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:09:59.846248 16955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:09:59.856086 16955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:09:59.857401 16955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:09:59.858702 16955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:09:59.870662 16960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:09:59.870544 16961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:09:59.870724 16963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:09:59.871284 16955 server_base.cc:1034] running on GCE node
I20260620 03:09:59.872045 16955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:09:59.873343 16955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:09:59.874588 16955 hybrid_clock.cc:648] HybridClock initialized: now 1781924999874522 us; error 39 us; skew 500 ppm
I20260620 03:09:59.874902 16955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:09:59 dist-test-slave-xv68 krb5kdc[15878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:09:59.878031 16955 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:09:59.879437 16955 webserver.cc:460] Webserver started at http://127.0.97.62:44111/ using document root <none> and password file <none>
I20260620 03:09:59.880061 16955 fs_manager.cc:359] Metadata directory not provided
I20260620 03:09:59.880155 16955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:09:59.885360 16955 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.001s
I20260620 03:09:59.888032 16970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:09:59.889293 16955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:09:59.889427 16955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "dc34559247524fc28ef40ba4961a7aa0"
format_stamp: "Formatted at 2026-06-20 03:09:13 on dist-test-slave-xv68"
server_key: "261bf68e77cbbbdc47b7f15303eef437"
server_key_iv: "c2d074a369ce2f207b8bdee860ff81f2"
server_key_version: "encryptionkey@0"
I20260620 03:09:59.890352 16955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:09:59.911703 16955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:09:59.915678 16955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:09:59.915916 16955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:09:59.934746 16955 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38739
I20260620 03:09:59.934842 17021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38739 every 8 connection(s)
I20260620 03:09:59.936302 16955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:09:59.942359 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 16955
I20260620 03:09:59.943428 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40843
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38739
--webserver_port=37357
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:09:59.943694 17022 sys_catalog.cc:263] Verifying existing consensus state
I20260620 03:09:59.948899 17022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: Bootstrap starting.
I20260620 03:09:59.969883 17022 log.cc:824] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: Log is configured to *not* fsync() on all Append() calls
I20260620 03:09:59.985643 17022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 03:09:59.986196 17022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: Bootstrap complete.
I20260620 03:09:59.995886 17022 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc34559247524fc28ef40ba4961a7aa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38739 } }
I20260620 03:09:59.996832 17022 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc34559247524fc28ef40ba4961a7aa0, State: Initialized, Role: FOLLOWER
I20260620 03:09:59.997546 17022 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc34559247524fc28ef40ba4961a7aa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38739 } }
I20260620 03:09:59.997790 17022 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:09:59.997902 17022 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:09:59.998095 17022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 2 FOLLOWER]: Advancing to term 3
I20260620 03:10:00.001976 17022 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc34559247524fc28ef40ba4961a7aa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38739 } }
I20260620 03:10:00.002523 17022 leader_election.cc:302] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc34559247524fc28ef40ba4961a7aa0; no voters:
I20260620 03:10:00.003232 17022 leader_election.cc:288] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 03:10:00.003394 17026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 03:10:00.004710 17026 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [term 3 LEADER]: Becoming Leader. State: Replica: dc34559247524fc28ef40ba4961a7aa0, State: Running, Role: LEADER
I20260620 03:10:00.005291 17026 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc34559247524fc28ef40ba4961a7aa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38739 } }
I20260620 03:10:00.005585 17022 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:10:00.007843 17027 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "dc34559247524fc28ef40ba4961a7aa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc34559247524fc28ef40ba4961a7aa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38739 } } }
I20260620 03:10:00.008258 17027 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:00.010540 17028 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc34559247524fc28ef40ba4961a7aa0. Latest consensus state: current_term: 3 leader_uuid: "dc34559247524fc28ef40ba4961a7aa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc34559247524fc28ef40ba4961a7aa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38739 } } }
I20260620 03:10:00.010974 17028 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:00.012789 17032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:10:00.022257 17032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:10:00.024052 17032 catalog_manager.cc:1259] Loaded cluster ID: 56bb5e45351a49d884f122283afcd123
I20260620 03:10:00.024235 17032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:10:00.028807 17032 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:10:00.029021 17043 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 03:10:00.031483 17032 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: Loaded TSK: 0
I20260620 03:10:00.031684 17032 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dc34559247524fc28ef40ba4961a7aa0: Loaded TSK: 1
F20260620 03:10:00.033087 17032 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f13d77ab9ad google::LogMessage::Fail() at ??:0
@ 0x7f13d77ae88a google::LogMessage::SendToLog() at ??:0
@ 0x7f13d77ab4de google::LogMessage::Flush() at ??:0
@ 0x7f13d77aca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f13db50f7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f13db4fe73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f13ed1da1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f13ed1d9783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f13ed2652a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f13ed2650c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f13ed2af5e9 std::function<>::operator()() at ??:0
@ 0x7f13ed1ddcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f13ed1dbc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f13ed263610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f13ed263439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f13da3f7345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f13da3fa91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f13da3fa799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f13da3d5eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f13dd6af6db start_thread at ??:0
@ 0x7f13d664561f clone at ??:0
W20260620 03:10:00.332894 17024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:00.333168 17024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:00.333325 17024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:00.340723 17024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:00.340848 17024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:00.341019 17024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:10:00.350945 17024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37357
--tserver_master_addrs=127.0.97.62:38739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:00.352218 17024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:00.353485 17024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:00.366638 17070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:00.366736 17069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:00.366931 17072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:00.367879 17024 server_base.cc:1034] running on GCE node
I20260620 03:10:00.368544 17024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:00.369397 17024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:00.370641 17024 hybrid_clock.cc:648] HybridClock initialized: now 1781925000370598 us; error 56 us; skew 500 ppm
I20260620 03:10:00.370949 17024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:00 dist-test-slave-xv68 krb5kdc[15878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:00.373914 17024 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:10:00.375249 17024 webserver.cc:460] Webserver started at http://127.0.97.1:37357/ using document root <none> and password file <none>
I20260620 03:10:00.375924 17024 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:00.376072 17024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:00.381443 17024 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 03:10:00.384133 17079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:00.385404 17024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:10:00.385576 17024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "deedccf28293476ebd2894c104fb6372"
format_stamp: "Formatted at 2026-06-20 03:09:14 on dist-test-slave-xv68"
server_key: "ba2a3f9b6493c6f14d950ab98e74cd8e"
server_key_iv: "f8829fb2d1300b2302915e106dda1048"
server_key_version: "encryptionkey@0"
I20260620 03:10:00.386561 17024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:00.411456 17024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:00.416114 17024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:00.416404 17024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:00.417440 17024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:00.418756 17024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:00.418854 17024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:00.419003 17024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:00.419085 17024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:00.452055 17024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40843
I20260620 03:10:00.452176 17191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40843 every 8 connection(s)
I20260620 03:10:00.453522 17024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:10:00.458184 17194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:00.456424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:39283)
0620 03:10:00.457171 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:00.457387 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":285,"threads_started":1}
W20260620 03:10:00.458798 17192 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38739 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38739: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
I20260620 03:10:00.459640 17194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:00.459265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:34587)
0620 03:10:00.459474 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:00.459540 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260620 03:10:00.460290 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 17024
I20260620 03:10:00.460484 17194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:00.460128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:59405)
0620 03:10:00.460263 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:00.460340 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 03:10:00.460727 17192 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 03:10:00.461000 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33619
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38739
--webserver_port=40109
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:00.787689 17195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:00.787961 17195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:00.788168 17195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:00.795746 17195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:00.795868 17195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:00.796053 17195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:10:00.806075 17195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40109
--tserver_master_addrs=127.0.97.62:38739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:00.807438 17195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:00.808744 17195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:00.821406 17201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:00.821449 17203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:00.821693 17200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:00.822614 17195 server_base.cc:1034] running on GCE node
I20260620 03:10:00.823396 17195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:00.824415 17195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:00.825647 17195 hybrid_clock.cc:648] HybridClock initialized: now 1781925000825591 us; error 53 us; skew 500 ppm
I20260620 03:10:00.825973 17195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:00 dist-test-slave-xv68 krb5kdc[15878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:00.829289 17195 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:10:00.830734 17195 webserver.cc:460] Webserver started at http://127.0.97.2:40109/ using document root <none> and password file <none>
I20260620 03:10:00.831503 17195 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:00.831655 17195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:00.837056 17195 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 03:10:00.839896 17210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:00.841326 17195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:10:00.841521 17195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "c495ad6c2e444228b91cddf586a37693"
format_stamp: "Formatted at 2026-06-20 03:09:15 on dist-test-slave-xv68"
server_key: "5124765e3bfbda5157b26fc791a49903"
server_key_iv: "2b477e463df3aec3fb93c44376025421"
server_key_version: "encryptionkey@0"
I20260620 03:10:00.842515 17195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:00.861020 17195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:00.865221 17195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:00.865504 17195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:00.866441 17195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:00.867770 17195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:00.867889 17195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:00.868031 17195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:00.868119 17195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:00.900735 17195 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33619
I20260620 03:10:00.900820 17322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33619 every 8 connection(s)
I20260620 03:10:00.902225 17195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:10:00.903347 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 17195
I20260620 03:10:00.904085 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39969
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38739
--webserver_port=43827
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:10:00.907579 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:00.905578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:44565)
0620 03:10:00.906358 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:00.906539 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":299,"threads_started":1}
W20260620 03:10:00.908341 17323 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38739 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38739: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
I20260620 03:10:00.909271 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:00.908901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:34561)
0620 03:10:00.909097 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:00.909165 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 03:10:00.910252 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:00.909917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:35515)
0620 03:10:00.910046 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:00.910146 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 03:10:00.910544 17323 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 03:10:01.229853 17326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:01.230140 17326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:01.230237 17326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:01.237603 17326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:01.237725 17326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:01.237846 17326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:10:01.247767 17326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43827
--tserver_master_addrs=127.0.97.62:38739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:01.248994 17326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:01.250211 17326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:01.263533 17331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:01.263506 17334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:01.263485 17332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:01.264312 17326 server_base.cc:1034] running on GCE node
I20260620 03:10:01.264930 17326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:01.265811 17326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:01.267026 17326 hybrid_clock.cc:648] HybridClock initialized: now 1781925001266977 us; error 42 us; skew 500 ppm
I20260620 03:10:01.267345 17326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:01 dist-test-slave-xv68 krb5kdc[15878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:01.270351 17326 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:10:01.271919 17326 webserver.cc:460] Webserver started at http://127.0.97.3:43827/ using document root <none> and password file <none>
I20260620 03:10:01.272583 17326 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:01.272710 17326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:01.277956 17326 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 03:10:01.280786 17341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:01.282172 17326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:10:01.282347 17326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "82eb0c49468442e7b4b6f4029cf899c3"
format_stamp: "Formatted at 2026-06-20 03:09:15 on dist-test-slave-xv68"
server_key: "fb6af908d8a3d3f49a8d1710936d65f5"
server_key_iv: "d02459d083c75280e7378b4ee36405b2"
server_key_version: "encryptionkey@0"
I20260620 03:10:01.283281 17326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:01.310959 17326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:01.315604 17326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:01.315891 17326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:01.316910 17326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:01.318215 17326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:01.318297 17326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:01.318449 17326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:01.318552 17326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:01.351552 17326 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39969
I20260620 03:10:01.351612 17453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39969 every 8 connection(s)
I20260620 03:10:01.353035 17326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:10:01.357717 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:01.355897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:45373)
0620 03:10:01.356664 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:01.356871 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":330,"threads_started":1}
W20260620 03:10:01.358354 17454 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38739 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38739: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
I20260620 03:10:01.359157 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:01.358859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:52729)
0620 03:10:01.358993 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:01.359057 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 03:10:01.359758 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 17326
I20260620 03:10:01.360121 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:01.359776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:59715)
0620 03:10:01.359950 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:01.360019 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 03:10:01.360414 17454 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 03:10:01.462071 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:01.461234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:42741)
0620 03:10:01.461767 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:01.461856 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":250,"threads_started":1}
I20260620 03:10:01.911942 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:01.911135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:41877)
0620 03:10:01.911640 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:01.911723 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":284,"threads_started":1}
I20260620 03:10:02.361577 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:02.360860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:55779)
0620 03:10:02.361279 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:02.361360 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":195,"threads_started":1}
I20260620 03:10:02.463670 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:02.462936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:45293)
0620 03:10:02.463421 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:02.463501 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":212,"threads_started":1}
I20260620 03:10:02.913820 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:02.912942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:54089)
0620 03:10:02.913491 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:02.913613 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":299,"threads_started":1}
I20260620 03:10:03.363133 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:03.362387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:43751)
0620 03:10:03.362889 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:03.362964 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":282,"threads_started":1}
I20260620 03:10:03.465269 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:03.464608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:33827)
0620 03:10:03.465027 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:03.465100 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":208,"threads_started":1}
I20260620 03:10:03.915501 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:03.914745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:54555)
0620 03:10:03.915182 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:03.915264 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":228,"threads_started":1}
I20260620 03:10:04.364506 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:04.363898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:56575)
0620 03:10:04.364280 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:04.364349 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":198,"threads_started":1}
I20260620 03:10:04.466838 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:04.466155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:42103)
0620 03:10:04.466588 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:04.466661 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":235,"threads_started":1}
I20260620 03:10:04.917114 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:04.916399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:36977)
0620 03:10:04.916832 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:04.916911 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":216,"threads_started":1}
I20260620 03:10:05.366066 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:05.365361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:56035)
0620 03:10:05.365818 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:05.365897 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":240,"threads_started":1}
I20260620 03:10:05.468568 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:05.467808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:52951)
0620 03:10:05.468328 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:05.468401 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":307,"threads_started":1}
I20260620 03:10:05.918805 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:05.918058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:51681)
0620 03:10:05.918539 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:05.918619 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":261,"threads_started":1}
I20260620 03:10:06.367765 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:06.367024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:50037)
0620 03:10:06.367522 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:06.367597 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":305,"threads_started":1}
I20260620 03:10:06.470216 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:06.469416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:53463)
0620 03:10:06.469907 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:06.470016 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":271,"threads_started":1}
I20260620 03:10:06.920902 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:06.919995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:36627)
0620 03:10:06.920521 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:06.920646 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":285,"threads_started":1}
I20260620 03:10:07.369750 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:07.368848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:46687)
0620 03:10:07.369429 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:07.369557 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":311,"threads_started":1}
I20260620 03:10:07.472263 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:07.471226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:39227)
0620 03:10:07.471918 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:07.472003 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":268,"threads_started":1}
I20260620 03:10:07.923235 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:07.922024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:34449)
0620 03:10:07.922750 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:07.922832 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":460,"threads_started":1}
I20260620 03:10:08.371852 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:08.370904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:55221)
0620 03:10:08.371547 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:08.371634 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":279,"threads_started":1}
I20260620 03:10:08.473991 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:08.473213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:58509)
0620 03:10:08.473708 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:08.473802 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":258,"threads_started":1}
I20260620 03:10:08.925251 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:08.924465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:56317)
0620 03:10:08.924989 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:08.925082 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":273,"threads_started":1}
I20260620 03:10:09.373639 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:09.372828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:40435)
0620 03:10:09.373357 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:09.373466 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":252,"threads_started":1}
I20260620 03:10:09.475761 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:09.474938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:33435)
0620 03:10:09.475481 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:09.475575 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":229,"threads_started":1}
I20260620 03:10:09.927507 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:09.926463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:40005)
0620 03:10:09.927023 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:09.927118 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":290,"threads_started":1}
I20260620 03:10:10.375787 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:10.374842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:48241)
0620 03:10:10.375505 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:10.375597 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":247,"threads_started":1}
I20260620 03:10:10.477763 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:10.476923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:47603)
0620 03:10:10.477502 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:10.477590 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":312,"threads_started":1}
I20260620 03:10:10.929524 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:10.928634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:40561)
0620 03:10:10.929274 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:10.929361 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":336,"threads_started":1}
I20260620 03:10:11.377542 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:11.376811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:49511)
0620 03:10:11.377261 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:11.377376 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":226,"threads_started":1}
I20260620 03:10:11.479599 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:11.478826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:51633)
0620 03:10:11.479247 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:11.479405 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":206,"threads_started":1}
I20260620 03:10:11.931713 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:11.930742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:59495)
0620 03:10:11.931422 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:11.931525 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":356,"threads_started":1}
I20260620 03:10:12.379487 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:12.378609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:53705)
0620 03:10:12.379083 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:12.379176 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":244,"threads_started":1}
I20260620 03:10:12.481334 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:12.480517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:60415)
0620 03:10:12.481034 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:12.481127 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":273,"threads_started":1}
I20260620 03:10:12.933771 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:12.932845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:45647)
0620 03:10:12.933432 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:12.933543 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":299,"threads_started":1}
I20260620 03:10:13.381115 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:13.380423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:59047)
0620 03:10:13.380863 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:13.380950 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":221,"threads_started":1}
I20260620 03:10:13.483047 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:13.482250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:36345)
0620 03:10:13.482752 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:13.482839 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":269,"threads_started":1}
I20260620 03:10:13.935819 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:13.934890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:50799)
0620 03:10:13.935446 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:13.935582 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":225,"threads_started":1}
I20260620 03:10:14.382841 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:14.382099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:54633)
0620 03:10:14.382571 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:14.382657 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":235,"threads_started":1}
I20260620 03:10:14.484755 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:14.484009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:45813)
0620 03:10:14.484491 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:14.484569 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":233,"threads_started":1}
I20260620 03:10:14.937657 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:14.936844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:36697)
0620 03:10:14.937326 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:14.937417 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":229,"threads_started":1}
I20260620 03:10:15.384531 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:15.383825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:39435)
0620 03:10:15.384240 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:15.384340 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":212,"threads_started":1}
I20260620 03:10:15.486919 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:15.486035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.1:49213)
0620 03:10:15.486639 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:15.486714 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":288,"threads_started":1}
I20260620 03:10:15.939862 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:15.938922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.2:42779)
0620 03:10:15.939550 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:15.939647 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":248,"threads_started":1}
I20260620 03:10:16.368726 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 17024
I20260620 03:10:16.386637 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:16.385707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38739 (local address 127.0.97.3:50107)
0620 03:10:16.386247 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:16.386353 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":256,"threads_started":1}
I20260620 03:10:16.393254 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:16.369863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.0.97.1:40843)
0620 03:10:16.370536 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:16.370556 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:16.370600 (+ 44us) server_negotiation.cc:407] Connection header received
0620 03:10:16.372535 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:16.372544 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:16.372961 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:16.373248 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:16.374555 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:16.375884 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:16.377161 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:16.377867 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:16.378006 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:16.386341 (+ 8335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:16.386353 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:16.386378 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:16.386432 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:16.389531 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:16.390287 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:16.390295 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:16.390302 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:16.390561 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:16.390992 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:16.390996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:16.390999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:16.391671 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:16.392031 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:16.392517 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 03:10:16.392969 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":286,"threads_started":1}
I20260620 03:10:16.394902 17187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:16.498440 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 17024
I20260620 03:10:16.515748 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 17195
I20260620 03:10:16.533743 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:16.516784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50034 (local address 127.0.97.2:33619)
0620 03:10:16.517342 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:16.517356 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:16.517393 (+ 37us) server_negotiation.cc:407] Connection header received
0620 03:10:16.518926 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:16.518936 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:16.519428 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:16.519731 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:16.521035 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:16.522259 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:16.523681 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:16.524444 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:16.524598 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:16.527226 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:16.527235 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:16.527250 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:16.527415 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:16.530349 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:16.531122 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:16.531130 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:16.531137 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:16.531425 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:16.531839 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:16.531843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:16.531845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:16.532416 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:16.532644 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:16.533080 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 03:10:16.533459 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":269,"threads_started":1}
I20260620 03:10:16.535115 17318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:16.617866 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 17195
I20260620 03:10:16.635320 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 17326
I20260620 03:10:16.653191 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:16.636367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.0.97.3:39969)
0620 03:10:16.636922 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:16.636936 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:16.636978 (+ 42us) server_negotiation.cc:407] Connection header received
0620 03:10:16.638572 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:16.638583 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:16.638993 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:16.639440 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:16.640846 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:16.642103 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:16.643545 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:16.644176 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:16.644361 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:16.646901 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:16.646909 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:16.646923 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:16.646958 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:16.649947 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:16.650617 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:16.650627 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:16.650639 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:16.650798 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:16.651166 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:16.651172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:16.651175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:16.651742 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:16.652046 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:16.652498 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 03:10:16.652886 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":257,"threads_started":1}
I20260620 03:10:16.654567 17449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:16.737736 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 17326
2026-06-20T03:10:16Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64534 ms)
[----------] 30 tests from SecurityITest (296330 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:10:16 dist-test-slave-xv68 krb5kdc[17512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:10:16 dist-test-slave-xv68 krb5kdc[17512](info): set up 2 sockets
Jun 20 03:10:16 dist-test-slave-xv68 krb5kdc[17512](info): commencing operation
krb5kdc: starting...
W20260620 03:10:17.859342 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:10:18 dist-test-slave-xv68 krb5kdc[17512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:10:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:10:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:10:18.102406 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42815
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:18.429193 17525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:18.429499 17525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:18.429661 17525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:18.436645 17525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:10:18.436739 17525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:18.436849 17525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:18.436950 17525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:10:18.437036 17525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:10:18.446605 17525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:18.447961 17525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:18.449329 17525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:18.461575 17530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:18.461447 17531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:18.461413 17533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:18.461948 17525 server_base.cc:1034] running on GCE node
I20260620 03:10:18.462998 17525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:18.464439 17525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:18.465689 17525 hybrid_clock.cc:648] HybridClock initialized: now 1781925018465643 us; error 35 us; skew 500 ppm
I20260620 03:10:18.466037 17525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:18 dist-test-slave-xv68 krb5kdc[17512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:18.469111 17525 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:10:18.470548 17525 webserver.cc:460] Webserver started at http://127.0.97.62:36015/ using document root <none> and password file <none>
I20260620 03:10:18.471160 17525 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:18.471256 17525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:18.471599 17525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:18.474411 17525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "65d28980cb524366b88e80d8f81596bb"
format_stamp: "Formatted at 2026-06-20 03:10:18 on dist-test-slave-xv68"
server_key: "0129fc55bf189d413aa0d9e78c847e74"
server_key_iv: "eee92f32126b8f096fd4fd53cc064390"
server_key_version: "encryptionkey@0"
I20260620 03:10:18.475099 17525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "65d28980cb524366b88e80d8f81596bb"
format_stamp: "Formatted at 2026-06-20 03:10:18 on dist-test-slave-xv68"
server_key: "0129fc55bf189d413aa0d9e78c847e74"
server_key_iv: "eee92f32126b8f096fd4fd53cc064390"
server_key_version: "encryptionkey@0"
I20260620 03:10:18.480028 17525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 03:10:18.483182 17540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:18.484463 17525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:10:18.484619 17525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "65d28980cb524366b88e80d8f81596bb"
format_stamp: "Formatted at 2026-06-20 03:10:18 on dist-test-slave-xv68"
server_key: "0129fc55bf189d413aa0d9e78c847e74"
server_key_iv: "eee92f32126b8f096fd4fd53cc064390"
server_key_version: "encryptionkey@0"
I20260620 03:10:18.484795 17525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:18.511969 17525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:18.515956 17525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:18.516189 17525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:18.535104 17525 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42815
I20260620 03:10:18.535151 17591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42815 every 8 connection(s)
I20260620 03:10:18.536513 17525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:10:18.540311 17592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:18.545069 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 17525
I20260620 03:10:18.545284 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:10:18.546141 388 external_mini_cluster.cc:1428] Setting key 2b03d67f9532b76b108af3cda6ae545e
I20260620 03:10:18.551806 17592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb: Bootstrap starting.
I20260620 03:10:18.555367 17592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:18.556617 17592 log.cc:824] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:18.560036 17592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb: No bootstrap required, opened a new log
Jun 20 03:10:18 dist-test-slave-xv68 krb5kdc[17512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:10:18.568080 17592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d28980cb524366b88e80d8f81596bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42815 } }
I20260620 03:10:18.568437 17592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:18.568598 17592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d28980cb524366b88e80d8f81596bb, State: Initialized, Role: FOLLOWER
I20260620 03:10:18.569271 17592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d28980cb524366b88e80d8f81596bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42815 } }
I20260620 03:10:18.569568 17592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:10:18.569741 17592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:10:18.569922 17592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:18.570816 17595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:18.548480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.0.97.62:42815)
0620 03:10:18.549331 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:18.549342 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:18.549383 (+ 41us) server_negotiation.cc:407] Connection header received
0620 03:10:18.551070 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:18.551090 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:18.551519 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:18.551825 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:18.553639 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:18.554960 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:18.556955 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:18.558084 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:18.558211 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:18.562971 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:18.562978 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:18.562995 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:18.563032 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:18.566355 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:18.567102 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:18.567110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:18.567120 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:18.567360 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:18.567773 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:18.567779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:18.567782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:18.568314 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:18.568658 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:18.569123 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 03:10:18.569634 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":417,"threads_started":1}
I20260620 03:10:18.573132 17592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d28980cb524366b88e80d8f81596bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42815 } }
I20260620 03:10:18.573721 17592 leader_election.cc:302] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65d28980cb524366b88e80d8f81596bb; no voters:
I20260620 03:10:18.574316 17592 leader_election.cc:288] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:10:18.574455 17597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:18.575117 17597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb [term 1 LEADER]: Becoming Leader. State: Replica: 65d28980cb524366b88e80d8f81596bb, State: Running, Role: LEADER
I20260620 03:10:18.575649 17597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d28980cb524366b88e80d8f81596bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42815 } }
I20260620 03:10:18.576357 17592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:10:18.578476 17598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65d28980cb524366b88e80d8f81596bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d28980cb524366b88e80d8f81596bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42815 } } }
I20260620 03:10:18.578816 17598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:18.579526 17599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65d28980cb524366b88e80d8f81596bb. Latest consensus state: current_term: 1 leader_uuid: "65d28980cb524366b88e80d8f81596bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d28980cb524366b88e80d8f81596bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42815 } } }
I20260620 03:10:18.579867 17599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:18.580117 17602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:10:18.586894 17602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:10:18.598470 17602 catalog_manager.cc:1347] Generated new cluster ID: 45e6b3209e2941a5af59f52a4bce0f1c
I20260620 03:10:18.598667 17602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:10:18.615376 17602 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:10:18.616722 17602 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:10:18.627835 17602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb: Generated new TSK 0
I20260620 03:10:18.628657 17602 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:10:18.697310 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42815
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:19.029836 17618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:19.030102 17618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:19.030193 17618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:19.037662 17618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:19.037782 17618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:19.037897 17618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:10:19.047801 17618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:19.049351 17618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:19.050686 17618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:19.063800 17624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:19.063832 17623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:19.064035 17626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:19.064618 17618 server_base.cc:1034] running on GCE node
I20260620 03:10:19.065202 17618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:19.066133 17618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:19.067435 17618 hybrid_clock.cc:648] HybridClock initialized: now 1781925019067373 us; error 60 us; skew 500 ppm
I20260620 03:10:19.067802 17618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:19 dist-test-slave-xv68 krb5kdc[17512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:19.071062 17618 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:10:19.072614 17618 webserver.cc:460] Webserver started at http://127.0.97.1:42491/ using document root <none> and password file <none>
I20260620 03:10:19.073283 17618 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:19.073426 17618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:19.073740 17618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:19.076843 17618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "86d47f46a8ce4a93a2e3f8591dcdb2ad"
format_stamp: "Formatted at 2026-06-20 03:10:19 on dist-test-slave-xv68"
server_key: "2bee6eee18b47d2594e1f60e20fa6943"
server_key_iv: "6015705bce2a02db2912ea8ea30d97a0"
server_key_version: "encryptionkey@0"
I20260620 03:10:19.077696 17618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "86d47f46a8ce4a93a2e3f8591dcdb2ad"
format_stamp: "Formatted at 2026-06-20 03:10:19 on dist-test-slave-xv68"
server_key: "2bee6eee18b47d2594e1f60e20fa6943"
server_key_iv: "6015705bce2a02db2912ea8ea30d97a0"
server_key_version: "encryptionkey@0"
I20260620 03:10:19.082656 17618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 03:10:19.086493 17633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:19.087976 17618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 03:10:19.088157 17618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "86d47f46a8ce4a93a2e3f8591dcdb2ad"
format_stamp: "Formatted at 2026-06-20 03:10:19 on dist-test-slave-xv68"
server_key: "2bee6eee18b47d2594e1f60e20fa6943"
server_key_iv: "6015705bce2a02db2912ea8ea30d97a0"
server_key_version: "encryptionkey@0"
I20260620 03:10:19.088352 17618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:19.100229 17618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:19.104612 17618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:19.104920 17618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:19.105918 17618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:19.107225 17618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:19.107365 17618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:19.107539 17618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:19.107610 17618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:19.139818 17618 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35575
I20260620 03:10:19.139900 17745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35575 every 8 connection(s)
I20260620 03:10:19.141040 17618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:10:19.145540 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 17618
I20260620 03:10:19.145748 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:10:19.146549 388 external_mini_cluster.cc:1428] Setting key 01c444c4329e570fbecbdc240ad04369
Jun 20 03:10:19 dist-test-slave-xv68 krb5kdc[17512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:10:19.165047 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:19.143580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47147 (local address 127.0.97.62:42815)
0620 03:10:19.144017 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:19.144021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:19.144944 (+ 923us) server_negotiation.cc:407] Connection header received
0620 03:10:19.147719 (+ 2775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:19.147730 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:19.147802 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:19.147890 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:19.149830 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:19.150560 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:19.152983 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:19.153367 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:19.153464 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:19.157842 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:19.157850 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:19.157859 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:19.157925 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:19.161601 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:19.162384 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:19.162392 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:19.162399 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:19.162556 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:19.163030 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:19.163037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:19.163043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:19.163452 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:19.163602 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:19.164312 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 03:10:19.164568 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":215,"threads_started":1}
I20260620 03:10:19.165643 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:19.144102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42815 (local address 127.0.97.1:47147)
0620 03:10:19.144789 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:19.144833 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:19.147281 (+ 2448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:19.148157 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:19.148168 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:19.148865 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:10:19.149648 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:19.149675 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:19.150718 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:19.150726 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:19.152836 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:19.152857 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:19.153593 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:19.153601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:19.153909 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:19.154613 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:10:19.154649 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:10:19.157599 (+ 2950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:10:19.161765 (+ 4166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:10:19.161773 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:10:19.161785 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:10:19.162233 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:10:19.162695 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:10:19.162699 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:10:19.162702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:10:19.162896 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:10:19.163589 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:10:19.163595 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:10:19.163913 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 03:10:19.164145 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 03:10:19.164552 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":297,"threads_started":1}
I20260620 03:10:19.167230 17746 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42815
I20260620 03:10:19.167580 17746 heartbeater.cc:464] Registering TS with master...
I20260620 03:10:19.168273 17746 heartbeater.cc:511] Master 127.0.97.62:42815 requested a full tablet report, sending...
I20260620 03:10:19.170411 17557 ts_manager.cc:194] Registered new tserver with Master: 86d47f46a8ce4a93a2e3f8591dcdb2ad (127.0.97.1:35575)
I20260620 03:10:19.171893 17557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47147
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:10:19.204331 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42815
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:19.528366 17752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:19.528682 17752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:19.528855 17752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:19.536054 17752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:19.536194 17752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:19.536358 17752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:10:19.545881 17752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:19.547056 17752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:19.548359 17752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:19.561223 17758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:19.561741 17757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:19.561784 17760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:19.562263 17752 server_base.cc:1034] running on GCE node
I20260620 03:10:19.562886 17752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:19.563879 17752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:19.565127 17752 hybrid_clock.cc:648] HybridClock initialized: now 1781925019565083 us; error 52 us; skew 500 ppm
I20260620 03:10:19.565564 17752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:19 dist-test-slave-xv68 krb5kdc[17512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:19.568779 17752 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:10:19.570217 17752 webserver.cc:460] Webserver started at http://127.0.97.2:39975/ using document root <none> and password file <none>
I20260620 03:10:19.570845 17752 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:19.570948 17752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:19.571200 17752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:19.574038 17752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "8dd09c88b59d42faac181db73c78b5d2"
format_stamp: "Formatted at 2026-06-20 03:10:19 on dist-test-slave-xv68"
server_key: "f35b1142bec5647fb78e94afa66355f0"
server_key_iv: "55266a111893f3d800be017b15657af2"
server_key_version: "encryptionkey@0"
I20260620 03:10:19.574903 17752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8dd09c88b59d42faac181db73c78b5d2"
format_stamp: "Formatted at 2026-06-20 03:10:19 on dist-test-slave-xv68"
server_key: "f35b1142bec5647fb78e94afa66355f0"
server_key_iv: "55266a111893f3d800be017b15657af2"
server_key_version: "encryptionkey@0"
I20260620 03:10:19.580413 17752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 03:10:19.583822 17767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:19.585276 17752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 03:10:19.585427 17752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "8dd09c88b59d42faac181db73c78b5d2"
format_stamp: "Formatted at 2026-06-20 03:10:19 on dist-test-slave-xv68"
server_key: "f35b1142bec5647fb78e94afa66355f0"
server_key_iv: "55266a111893f3d800be017b15657af2"
server_key_version: "encryptionkey@0"
I20260620 03:10:19.585683 17752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:19.610054 17752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:19.613951 17752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:19.614239 17752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:19.615235 17752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:19.616608 17752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:19.616678 17752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:19.616849 17752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:19.616899 17752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:19.649763 17752 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46559
I20260620 03:10:19.649811 17879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46559 every 8 connection(s)
I20260620 03:10:19.651012 17752 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:10:19.657058 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 17752
I20260620 03:10:19.657270 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:10:19.657862 388 external_mini_cluster.cc:1428] Setting key d9713b6894ef4e559da4be858c497fda
Jun 20 03:10:19 dist-test-slave-xv68 krb5kdc[17512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:10:19.671263 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:19.653459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50051 (local address 127.0.97.62:42815)
0620 03:10:19.653679 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:19.653686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:19.654800 (+ 1114us) server_negotiation.cc:407] Connection header received
0620 03:10:19.656593 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:19.656597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:19.656668 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:19.656755 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:19.658804 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:19.659596 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:19.661089 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:19.661501 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:19.661635 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:19.665010 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:19.665019 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:19.665026 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:19.665072 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:19.668230 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:19.669119 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:19.669124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:19.669126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:19.669248 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:19.669704 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:19.669707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:19.669709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:19.669945 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:19.670053 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:19.670863 (+ 810us) server_negotiation.cc:299] Negotiation successful
0620 03:10:19.671028 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 03:10:19.672767 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:19.653932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42815 (local address 127.0.97.2:50051)
0620 03:10:19.654602 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:19.654675 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:19.656349 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:19.657030 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:19.657044 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:19.657833 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:10:19.658594 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:19.658644 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:19.659737 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:19.659743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:19.660936 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:19.660947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:19.661719 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:19.661726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:19.661941 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:19.662625 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:10:19.662645 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:10:19.664779 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:10:19.668364 (+ 3585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:10:19.668375 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:10:19.668393 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:10:19.668993 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:10:19.669352 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:10:19.669359 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:10:19.669366 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:10:19.669600 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:10:19.670075 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:10:19.670086 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:10:19.670495 (+ 409us) client_negotiation.cc:769] Sending connection context
0620 03:10:19.670795 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 03:10:19.671389 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":267,"threads_started":1}
I20260620 03:10:19.674230 17880 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42815
I20260620 03:10:19.674535 17880 heartbeater.cc:464] Registering TS with master...
I20260620 03:10:19.675233 17880 heartbeater.cc:511] Master 127.0.97.62:42815 requested a full tablet report, sending...
I20260620 03:10:19.676728 17557 ts_manager.cc:194] Registered new tserver with Master: 8dd09c88b59d42faac181db73c78b5d2 (127.0.97.2:46559)
I20260620 03:10:19.677632 17557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50051
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:10:19.712277 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42815
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:20.032948 17885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:20.033282 17885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:20.033456 17885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:20.040551 17885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:20.040701 17885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:20.040894 17885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:10:20.050616 17885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:20.051893 17885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:20.053229 17885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:20.066031 17891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:20.065936 17893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:20.066071 17890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:20.066725 17885 server_base.cc:1034] running on GCE node
I20260620 03:10:20.067283 17885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:20.068238 17885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:20.069459 17885 hybrid_clock.cc:648] HybridClock initialized: now 1781925020069419 us; error 54 us; skew 500 ppm
I20260620 03:10:20.069773 17885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:20 dist-test-slave-xv68 krb5kdc[17512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:20.072804 17885 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:10:20.074177 17885 webserver.cc:460] Webserver started at http://127.0.97.3:41875/ using document root <none> and password file <none>
I20260620 03:10:20.074827 17885 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:20.074923 17885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:20.075167 17885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:20.078143 17885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "2af99755cf3a4d05b5362f3f4299f456"
format_stamp: "Formatted at 2026-06-20 03:10:20 on dist-test-slave-xv68"
server_key: "2d342dd50a6ad7cf5f72d3bfecb02eab"
server_key_iv: "c5fd1808a2234aa35962c75b96b506df"
server_key_version: "encryptionkey@0"
I20260620 03:10:20.078846 17885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2af99755cf3a4d05b5362f3f4299f456"
format_stamp: "Formatted at 2026-06-20 03:10:20 on dist-test-slave-xv68"
server_key: "2d342dd50a6ad7cf5f72d3bfecb02eab"
server_key_iv: "c5fd1808a2234aa35962c75b96b506df"
server_key_version: "encryptionkey@0"
I20260620 03:10:20.083680 17885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260620 03:10:20.087088 17900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:20.088550 17885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:10:20.088718 17885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "2af99755cf3a4d05b5362f3f4299f456"
format_stamp: "Formatted at 2026-06-20 03:10:20 on dist-test-slave-xv68"
server_key: "2d342dd50a6ad7cf5f72d3bfecb02eab"
server_key_iv: "c5fd1808a2234aa35962c75b96b506df"
server_key_version: "encryptionkey@0"
I20260620 03:10:20.088910 17885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:20.106963 17885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:20.110777 17885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:20.111057 17885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:20.112030 17885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:20.113376 17885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:20.113478 17885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:20.113623 17885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:20.113694 17885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:20.146755 17885 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35515
I20260620 03:10:20.146777 18012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35515 every 8 connection(s)
I20260620 03:10:20.148068 17885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:10:20.154945 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 17885
I20260620 03:10:20.155138 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:10:20.155638 388 external_mini_cluster.cc:1428] Setting key 071e07ff2040fde57558f995c69a0481
Jun 20 03:10:20 dist-test-slave-xv68 krb5kdc[17512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:10:20.167080 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:20.150557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41841 (local address 127.0.97.62:42815)
0620 03:10:20.150733 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:20.150737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:20.151835 (+ 1098us) server_negotiation.cc:407] Connection header received
0620 03:10:20.153613 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:20.153619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:20.153703 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:20.153813 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:20.155271 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.156232 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.158028 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.158588 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.158709 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:20.161941 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:20.161946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:20.161949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:20.161981 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:20.164533 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:20.165173 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:20.165177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:20.165179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:20.165306 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:20.165697 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:20.165701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:20.165703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:20.165957 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:20.166075 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:20.166684 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 03:10:20.166843 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:10:20.168008 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:20.150966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42815 (local address 127.0.97.3:41841)
0620 03:10:20.151687 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:20.151729 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:20.153352 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:20.154037 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:20.154045 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:20.154528 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:10:20.155095 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:20.155119 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.156372 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.156377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:20.157882 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:20.157896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.158787 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.158792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:20.158979 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:20.159612 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:10:20.159633 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:10:20.161744 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:10:20.164667 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:10:20.164674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:10:20.164685 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:10:20.165075 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:10:20.165395 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:10:20.165398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:10:20.165400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:10:20.165586 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:10:20.166063 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:10:20.166069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:10:20.166348 (+ 279us) client_negotiation.cc:769] Sending connection context
0620 03:10:20.166584 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 03:10:20.166925 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":304,"threads_started":1}
I20260620 03:10:20.169515 18013 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42815
I20260620 03:10:20.169814 18013 heartbeater.cc:464] Registering TS with master...
I20260620 03:10:20.170500 18013 heartbeater.cc:511] Master 127.0.97.62:42815 requested a full tablet report, sending...
I20260620 03:10:20.172055 17557 ts_manager.cc:194] Registered new tserver with Master: 2af99755cf3a4d05b5362f3f4299f456 (127.0.97.3:35515)
I20260620 03:10:20.172806 17557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41841
I20260620 03:10:20.174679 17746 heartbeater.cc:503] Master 127.0.97.62:42815 was elected leader, sending a full tablet report...
I20260620 03:10:20.182945 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:10:20.200534 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:20.187555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.0.97.62:42815)
0620 03:10:20.187707 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:20.187711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:20.188194 (+ 483us) server_negotiation.cc:407] Connection header received
0620 03:10:20.188399 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:20.188403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:20.188463 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:20.188577 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:20.190016 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.190724 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.192380 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.192785 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.192916 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:20.195484 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:20.195489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:20.195492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:20.195527 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:20.197857 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:20.198525 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:20.198533 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:20.198540 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:20.198654 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:20.199021 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:20.199028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:20.199032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:20.199421 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:20.199692 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:20.200005 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 03:10:20.200224 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 03:10:20.204150 17557 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:10:20.211148 17557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:10:20.244797 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:20.223724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33772 (local address 127.0.97.3:35515)
0620 03:10:20.233914 (+ 10190us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:20.233926 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:20.236695 (+ 2769us) server_negotiation.cc:407] Connection header received
0620 03:10:20.236968 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:20.236975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:20.237193 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:20.237354 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:20.238284 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.239818 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.242241 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.243322 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.243459 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:20.243697 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:20.244263 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 03:10:20.244420 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9969,"thread_start_us":9811,"threads_started":1}
I20260620 03:10:20.244867 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:20.223234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35515 (local address 127.0.0.1:33772)
0620 03:10:20.236584 (+ 13350us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:20.236601 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:20.236795 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:20.237673 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:20.237677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:20.237699 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:20.238080 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:20.238093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.240464 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.240469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:20.242092 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:20.242127 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.243827 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.243833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:20.244027 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:20.244078 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 03:10:20.244230 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 03:10:20.244412 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13232,"mutex_wait_us":112,"thread_start_us":11521,"threads_started":1}
I20260620 03:10:20.245152 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:20.222910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43716 (local address 127.0.97.2:46559)
0620 03:10:20.236297 (+ 13387us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:20.236307 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:20.236339 (+ 32us) server_negotiation.cc:407] Connection header received
0620 03:10:20.236444 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:20.236452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:20.236719 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:20.237704 (+ 985us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:20.237959 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.239023 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.242316 (+ 3293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.243690 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.243798 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:20.243975 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:20.244673 (+ 698us) server_negotiation.cc:299] Negotiation successful
0620 03:10:20.244841 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13217,"thread_start_us":13102,"threads_started":1}
I20260620 03:10:20.245411 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:20.223620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.0.97.1:35575)
0620 03:10:20.232408 (+ 8788us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:20.232421 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:20.236303 (+ 3882us) server_negotiation.cc:407] Connection header received
0620 03:10:20.236419 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:20.236427 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:20.236686 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:20.236861 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:20.237777 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.239321 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.242185 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.243676 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.243798 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:20.244011 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:20.244946 (+ 935us) server_negotiation.cc:299] Negotiation successful
0620 03:10:20.245091 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8568,"thread_start_us":8397,"threads_started":1}
I20260620 03:10:20.245903 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:20.222846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35575 (local address 127.0.0.1:41314)
0620 03:10:20.235960 (+ 13114us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:20.235984 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:20.236143 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:20.237135 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:20.237139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:20.237165 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:20.237617 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:20.237628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.239470 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.239475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:20.242018 (+ 2543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:20.242034 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.244200 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.244205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:20.244357 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:20.244464 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 03:10:20.244564 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 03:10:20.244708 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12961,"mutex_wait_us":153,"thread_start_us":13492,"threads_started":1}
I20260620 03:10:20.245967 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:20.222726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46559 (local address 127.0.0.1:43716)
0620 03:10:20.234213 (+ 11487us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:20.234278 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:20.234513 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:20.236951 (+ 2438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:20.236958 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:20.236985 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:20.237458 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:20.237479 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.239155 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.239163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:20.242032 (+ 2869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:20.242044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.244162 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.244170 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:20.244351 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:20.244467 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 03:10:20.244557 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 03:10:20.245491 (+ 934us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11131,"mutex_wait_us":979,"thread_start_us":12915,"threads_started":1}
I20260620 03:10:20.248993 17948 tablet_service.cc:1467] Processing CreateTablet for tablet 1f07a365062e47c3aecb469cede5d9a5 (DEFAULT_TABLE table=test-table [id=64cfaabf6f1941ccb016c6f555a0571f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:10:20.249305 17815 tablet_service.cc:1467] Processing CreateTablet for tablet 1f07a365062e47c3aecb469cede5d9a5 (DEFAULT_TABLE table=test-table [id=64cfaabf6f1941ccb016c6f555a0571f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:10:20.249516 17681 tablet_service.cc:1467] Processing CreateTablet for tablet 1f07a365062e47c3aecb469cede5d9a5 (DEFAULT_TABLE table=test-table [id=64cfaabf6f1941ccb016c6f555a0571f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:10:20.250460 17815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f07a365062e47c3aecb469cede5d9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:20.250507 17948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f07a365062e47c3aecb469cede5d9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:20.250799 17681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f07a365062e47c3aecb469cede5d9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:20.262197 18031 tablet_bootstrap.cc:492] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2: Bootstrap starting.
I20260620 03:10:20.264578 18032 tablet_bootstrap.cc:492] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad: Bootstrap starting.
I20260620 03:10:20.266026 18031 tablet_bootstrap.cc:654] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:20.267469 18031 log.cc:824] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:20.267987 18033 tablet_bootstrap.cc:492] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456: Bootstrap starting.
I20260620 03:10:20.268079 18032 tablet_bootstrap.cc:654] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:20.269609 18032 log.cc:824] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:20.270821 18033 tablet_bootstrap.cc:654] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:20.271996 18033 log.cc:824] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:20.272840 18032 tablet_bootstrap.cc:492] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad: No bootstrap required, opened a new log
I20260620 03:10:20.272840 18031 tablet_bootstrap.cc:492] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2: No bootstrap required, opened a new log
I20260620 03:10:20.273113 18032 ts_tablet_manager.cc:1397] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 03:10:20.273134 18031 ts_tablet_manager.cc:1397] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 03:10:20.274833 18033 tablet_bootstrap.cc:492] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456: No bootstrap required, opened a new log
I20260620 03:10:20.275135 18033 ts_tablet_manager.cc:1397] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260620 03:10:20.280658 18031 raft_consensus.cc:348] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd09c88b59d42faac181db73c78b5d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46559 } } peers { permanent_uuid: "86d47f46a8ce4a93a2e3f8591dcdb2ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35575 } } peers { permanent_uuid: "2af99755cf3a4d05b5362f3f4299f456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35515 } }
I20260620 03:10:20.281008 18031 raft_consensus.cc:374] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:20.281103 18031 raft_consensus.cc:729] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dd09c88b59d42faac181db73c78b5d2, State: Initialized, Role: FOLLOWER
I20260620 03:10:20.281675 18031 consensus_queue.cc:260] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd09c88b59d42faac181db73c78b5d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46559 } } peers { permanent_uuid: "86d47f46a8ce4a93a2e3f8591dcdb2ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35575 } } peers { permanent_uuid: "2af99755cf3a4d05b5362f3f4299f456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35515 } }
I20260620 03:10:20.282673 17880 heartbeater.cc:503] Master 127.0.97.62:42815 was elected leader, sending a full tablet report...
I20260620 03:10:20.283651 18032 raft_consensus.cc:348] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd09c88b59d42faac181db73c78b5d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46559 } } peers { permanent_uuid: "86d47f46a8ce4a93a2e3f8591dcdb2ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35575 } } peers { permanent_uuid: "2af99755cf3a4d05b5362f3f4299f456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35515 } }
I20260620 03:10:20.284191 18032 raft_consensus.cc:374] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:20.284319 18032 raft_consensus.cc:729] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d47f46a8ce4a93a2e3f8591dcdb2ad, State: Initialized, Role: FOLLOWER
I20260620 03:10:20.284489 18031 ts_tablet_manager.cc:1428] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260620 03:10:20.284987 18032 consensus_queue.cc:260] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd09c88b59d42faac181db73c78b5d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46559 } } peers { permanent_uuid: "86d47f46a8ce4a93a2e3f8591dcdb2ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35575 } } peers { permanent_uuid: "2af99755cf3a4d05b5362f3f4299f456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35515 } }
I20260620 03:10:20.285450 18033 raft_consensus.cc:348] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd09c88b59d42faac181db73c78b5d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46559 } } peers { permanent_uuid: "86d47f46a8ce4a93a2e3f8591dcdb2ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35575 } } peers { permanent_uuid: "2af99755cf3a4d05b5362f3f4299f456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35515 } }
I20260620 03:10:20.286088 18033 raft_consensus.cc:374] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:20.286250 18033 raft_consensus.cc:729] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2af99755cf3a4d05b5362f3f4299f456, State: Initialized, Role: FOLLOWER
I20260620 03:10:20.286828 18033 consensus_queue.cc:260] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd09c88b59d42faac181db73c78b5d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46559 } } peers { permanent_uuid: "86d47f46a8ce4a93a2e3f8591dcdb2ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35575 } } peers { permanent_uuid: "2af99755cf3a4d05b5362f3f4299f456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35515 } }
I20260620 03:10:20.287492 18032 ts_tablet_manager.cc:1428] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 03:10:20.288007 18013 heartbeater.cc:503] Master 127.0.97.62:42815 was elected leader, sending a full tablet report...
I20260620 03:10:20.288439 18033 ts_tablet_manager.cc:1428] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
W20260620 03:10:20.393400 17747 tablet.cc:2369] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:10:20.399685 18014 tablet.cc:2369] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:10:20.402923 17881 tablet.cc:2369] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:10:20.585170 18037 raft_consensus.cc:482] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:10:20.585546 18037 raft_consensus.cc:504] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd09c88b59d42faac181db73c78b5d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46559 } } peers { permanent_uuid: "86d47f46a8ce4a93a2e3f8591dcdb2ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35575 } } peers { permanent_uuid: "2af99755cf3a4d05b5362f3f4299f456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35515 } }
I20260620 03:10:20.587564 18037 leader_election.cc:288] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86d47f46a8ce4a93a2e3f8591dcdb2ad (127.0.97.1:35575), 2af99755cf3a4d05b5362f3f4299f456 (127.0.97.3:35515)
I20260620 03:10:20.597564 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:20.588212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40791 (local address 127.0.97.1:35575)
0620 03:10:20.588442 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:20.588450 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:20.589124 (+ 674us) server_negotiation.cc:407] Connection header received
0620 03:10:20.589341 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:20.589346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:20.589417 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:20.589513 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:20.590432 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.591613 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.594785 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.596077 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.596218 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:20.596328 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:20.597130 (+ 802us) server_negotiation.cc:299] Negotiation successful
0620 03:10:20.597277 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 03:10:20.597612 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:20.588205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60735 (local address 127.0.97.3:35515)
0620 03:10:20.588434 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:20.588443 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:20.589184 (+ 741us) server_negotiation.cc:407] Connection header received
0620 03:10:20.589326 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:20.589331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:20.589410 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:20.589514 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:20.590481 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.591615 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.594784 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.596077 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.596218 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:20.596328 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:20.597130 (+ 802us) server_negotiation.cc:299] Negotiation successful
0620 03:10:20.597277 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 03:10:20.597918 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:20.587938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35515 (local address 127.0.97.2:60735)
0620 03:10:20.589108 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:20.589123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:20.589220 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:20.589598 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:20.589604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:20.589704 (+ 100us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:20.590318 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:20.590331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.591757 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.591763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:20.594612 (+ 2849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:20.594628 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.596407 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.596412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:20.596783 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:20.596927 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 03:10:20.597044 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 03:10:20.597267 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"mutex_wait_us":409,"spinlock_wait_cycles":19712,"thread_start_us":483,"threads_started":1}
I20260620 03:10:20.598044 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:20.587760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35575 (local address 127.0.97.2:40791)
0620 03:10:20.588987 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:20.589009 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:20.589169 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:20.589598 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:20.589604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:20.589638 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:20.590228 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:20.590250 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.591757 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.591763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:20.594611 (+ 2848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:20.594628 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.596405 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.596412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:20.596783 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:20.596927 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 03:10:20.597044 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 03:10:20.597267 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"mutex_wait_us":441,"thread_start_us":950,"threads_started":1}
I20260620 03:10:20.599224 17968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f07a365062e47c3aecb469cede5d9a5" candidate_uuid: "8dd09c88b59d42faac181db73c78b5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2af99755cf3a4d05b5362f3f4299f456" is_pre_election: true
I20260620 03:10:20.599225 17701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f07a365062e47c3aecb469cede5d9a5" candidate_uuid: "8dd09c88b59d42faac181db73c78b5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d47f46a8ce4a93a2e3f8591dcdb2ad" is_pre_election: true
I20260620 03:10:20.599849 17968 raft_consensus.cc:2454] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dd09c88b59d42faac181db73c78b5d2 in term 0.
I20260620 03:10:20.599846 17701 raft_consensus.cc:2454] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dd09c88b59d42faac181db73c78b5d2 in term 0.
I20260620 03:10:20.600622 17769 leader_election.cc:302] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2af99755cf3a4d05b5362f3f4299f456, 86d47f46a8ce4a93a2e3f8591dcdb2ad, 8dd09c88b59d42faac181db73c78b5d2; no voters:
I20260620 03:10:20.601128 18037 raft_consensus.cc:2781] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:10:20.601259 18037 raft_consensus.cc:482] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:10:20.601431 18037 raft_consensus.cc:3037] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:20.603761 18037 raft_consensus.cc:504] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd09c88b59d42faac181db73c78b5d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46559 } } peers { permanent_uuid: "86d47f46a8ce4a93a2e3f8591dcdb2ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35575 } } peers { permanent_uuid: "2af99755cf3a4d05b5362f3f4299f456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35515 } }
I20260620 03:10:20.604424 18037 leader_election.cc:288] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [CANDIDATE]: Term 1 election: Requested vote from peers 86d47f46a8ce4a93a2e3f8591dcdb2ad (127.0.97.1:35575), 2af99755cf3a4d05b5362f3f4299f456 (127.0.97.3:35515)
I20260620 03:10:20.604866 17968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f07a365062e47c3aecb469cede5d9a5" candidate_uuid: "8dd09c88b59d42faac181db73c78b5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2af99755cf3a4d05b5362f3f4299f456"
I20260620 03:10:20.604866 17701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f07a365062e47c3aecb469cede5d9a5" candidate_uuid: "8dd09c88b59d42faac181db73c78b5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d47f46a8ce4a93a2e3f8591dcdb2ad"
I20260620 03:10:20.605074 17968 raft_consensus.cc:3037] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:20.605089 17701 raft_consensus.cc:3037] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:20.607174 17968 raft_consensus.cc:2454] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dd09c88b59d42faac181db73c78b5d2 in term 1.
I20260620 03:10:20.607213 17701 raft_consensus.cc:2454] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dd09c88b59d42faac181db73c78b5d2 in term 1.
I20260620 03:10:20.607612 17769 leader_election.cc:302] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2af99755cf3a4d05b5362f3f4299f456, 8dd09c88b59d42faac181db73c78b5d2; no voters:
I20260620 03:10:20.607950 18037 raft_consensus.cc:2781] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:20.608415 18037 raft_consensus.cc:686] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [term 1 LEADER]: Becoming Leader. State: Replica: 8dd09c88b59d42faac181db73c78b5d2, State: Running, Role: LEADER
I20260620 03:10:20.608942 18037 consensus_queue.cc:237] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd09c88b59d42faac181db73c78b5d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46559 } } peers { permanent_uuid: "86d47f46a8ce4a93a2e3f8591dcdb2ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35575 } } peers { permanent_uuid: "2af99755cf3a4d05b5362f3f4299f456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35515 } }
I20260620 03:10:20.613354 17557 catalog_manager.cc:5495] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dd09c88b59d42faac181db73c78b5d2 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8dd09c88b59d42faac181db73c78b5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd09c88b59d42faac181db73c78b5d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86d47f46a8ce4a93a2e3f8591dcdb2ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2af99755cf3a4d05b5362f3f4299f456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35515 } health_report { overall_health: UNKNOWN } } }
I20260620 03:10:20.701123 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:20.695924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.0.97.2:46559)
0620 03:10:20.696116 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:20.696121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:20.696203 (+ 82us) server_negotiation.cc:407] Connection header received
0620 03:10:20.696373 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:20.696376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:20.696484 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:20.696606 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:10:20.696918 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.697652 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.699021 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.699486 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.699588 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:20.700087 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:10:20.700565 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:10:20.700675 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:20.700798 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 03:10:20.700860 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 03:10:20.708927 17968 raft_consensus.cc:1261] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456 [term 1 FOLLOWER]: Refusing update from remote peer 8dd09c88b59d42faac181db73c78b5d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:10:20.708925 17701 raft_consensus.cc:1261] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad [term 1 FOLLOWER]: Refusing update from remote peer 8dd09c88b59d42faac181db73c78b5d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:10:20.709862 18042 consensus_queue.cc:1035] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2af99755cf3a4d05b5362f3f4299f456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:10:20.710199 18037 consensus_queue.cc:1035] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86d47f46a8ce4a93a2e3f8591dcdb2ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:10:20.734355 17557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:10:20.734632 17557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:10:20.738531 17557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65d28980cb524366b88e80d8f81596bb: Sending DeleteTablet for 3 replicas of tablet 1f07a365062e47c3aecb469cede5d9a5
I20260620 03:10:20.739764 17815 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f07a365062e47c3aecb469cede5d9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:10:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43716
I20260620 03:10:20.739894 17948 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f07a365062e47c3aecb469cede5d9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:10:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33772
I20260620 03:10:20.740067 17681 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f07a365062e47c3aecb469cede5d9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:10:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41314
I20260620 03:10:20.742197 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 17618
I20260620 03:10:20.751883 18057 tablet_replica.cc:330] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad: stopping tablet replica
I20260620 03:10:20.752410 18057 raft_consensus.cc:2229] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:10:20.752736 18056 tablet_replica.cc:330] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456: stopping tablet replica
I20260620 03:10:20.752983 18057 raft_consensus.cc:2258] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:10:20.753126 18056 raft_consensus.cc:2229] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:10:20.753572 18056 raft_consensus.cc:2258] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:10:20.755887 18056 ts_tablet_manager.cc:1905] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:10:20.759222 18057 ts_tablet_manager.cc:1905] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:10:20.762838 18056 ts_tablet_manager.cc:1918] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:10:20.763013 18056 log.cc:1196] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/1f07a365062e47c3aecb469cede5d9a5
I20260620 03:10:20.763564 18056 ts_tablet_manager.cc:1939] T 1f07a365062e47c3aecb469cede5d9a5 P 2af99755cf3a4d05b5362f3f4299f456: Deleting consensus metadata
I20260620 03:10:20.765416 17542 catalog_manager.cc:4841] TS 2af99755cf3a4d05b5362f3f4299f456 (127.0.97.3:35515): tablet 1f07a365062e47c3aecb469cede5d9a5 (table test-table [id=64cfaabf6f1941ccb016c6f555a0571f]) successfully deleted
I20260620 03:10:20.765718 18055 tablet_replica.cc:330] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2: stopping tablet replica
I20260620 03:10:20.766172 18055 raft_consensus.cc:2229] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [term 1 LEADER]: Raft consensus shutting down.
Jun 20 03:10:20 dist-test-slave-xv68 krb5kdc[17512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:10:20.766753 18055 raft_consensus.cc:2258] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:10:20.767122 18057 ts_tablet_manager.cc:1918] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:10:20.767310 18057 log.cc:1196] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/1f07a365062e47c3aecb469cede5d9a5
I20260620 03:10:20.767922 18057 ts_tablet_manager.cc:1939] T 1f07a365062e47c3aecb469cede5d9a5 P 86d47f46a8ce4a93a2e3f8591dcdb2ad: Deleting consensus metadata
I20260620 03:10:20.769532 17541 catalog_manager.cc:4841] TS 86d47f46a8ce4a93a2e3f8591dcdb2ad (127.0.97.1:35575): tablet 1f07a365062e47c3aecb469cede5d9a5 (table test-table [id=64cfaabf6f1941ccb016c6f555a0571f]) successfully deleted
I20260620 03:10:20.770892 18055 ts_tablet_manager.cc:1905] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:10:20.774633 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:20.743929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41330 (local address 127.0.97.1:35575)
0620 03:10:20.744149 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:20.744159 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:20.752087 (+ 7928us) server_negotiation.cc:407] Connection header received
0620 03:10:20.752296 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:20.752305 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:20.752469 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:20.752606 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:20.755352 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.756865 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.759792 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.760470 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.760611 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:20.767428 (+ 6817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:20.767446 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:20.767474 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:20.767518 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:20.771080 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:20.772060 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:20.772070 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:20.772073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:20.772237 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:20.772717 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:20.772722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:20.772726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:20.773166 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:20.773475 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:20.773800 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 03:10:20.774010 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":66}
I20260620 03:10:20.775271 17741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:20.779489 18055 ts_tablet_manager.cc:1918] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:10:20.779671 18055 log.cc:1196] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/1f07a365062e47c3aecb469cede5d9a5
I20260620 03:10:20.780133 18055 ts_tablet_manager.cc:1939] T 1f07a365062e47c3aecb469cede5d9a5 P 8dd09c88b59d42faac181db73c78b5d2: Deleting consensus metadata
I20260620 03:10:20.781601 17544 catalog_manager.cc:4841] TS 8dd09c88b59d42faac181db73c78b5d2 (127.0.97.2:46559): tablet 1f07a365062e47c3aecb469cede5d9a5 (table test-table [id=64cfaabf6f1941ccb016c6f555a0571f]) successfully deleted
I20260620 03:10:20.888473 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 17618
I20260620 03:10:20.912269 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 17752
Jun 20 03:10:20 dist-test-slave-xv68 krb5kdc[17512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:10:20.930032 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:20.913094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.0.97.2:46559)
0620 03:10:20.913271 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:20.913275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:20.913293 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:10:20.913530 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:20.913534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:20.913609 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:20.913691 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:20.915253 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.916088 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.918055 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:20.918490 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:20.918581 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:20.923846 (+ 5265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:20.923859 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:20.923875 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:20.923920 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:20.927027 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:20.927998 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:20.928004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:20.928007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:20.928154 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:20.928601 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:20.928605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:20.928607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:20.928969 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:20.929222 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:20.929529 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 03:10:20.929694 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 03:10:20.930879 17875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:21.043237 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 17752
I20260620 03:10:21.061730 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 17885
Jun 20 03:10:21 dist-test-slave-xv68 krb5kdc[17512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:10:21.077337 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:21.062547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.0.97.3:35515)
0620 03:10:21.062721 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:21.062726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:21.062744 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:10:21.062821 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:21.062825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:21.062888 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:21.062979 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:21.064268 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:21.065064 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:21.066386 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:21.066840 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:21.066949 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:21.071580 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:21.071594 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:21.071610 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:21.071648 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:21.074590 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:21.075200 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:21.075206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:21.075209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:21.075546 (+ 337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:21.075882 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:21.075885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:21.075887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:21.076223 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:21.076500 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:21.076816 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 03:10:21.077006 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:10:21.078084 18008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:21.166639 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 17885
I20260620 03:10:21.183730 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 17525
I20260620 03:10:21.184886 17587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:21.294328 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 17525
2026-06-20T03:10:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4559 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T03:10:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:10:21Z Disabled control of system clock
I20260620 03:10:21.370041 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40749
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 03:10:21.695459 18066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:21.695735 18066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:21.695900 18066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:21.702970 18066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:10:21.703069 18066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:21.703152 18066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:21.703239 18066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:10:21.703342 18066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:10:21.712468 18066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40749
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:21.713706 18066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:21.715484 18066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:21.726709 18071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:21.726758 18072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:21.726737 18074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:21.728014 18066 server_base.cc:1034] running on GCE node
I20260620 03:10:21.728808 18066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:21.730132 18066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:21.731422 18066 hybrid_clock.cc:648] HybridClock initialized: now 1781925021731354 us; error 56 us; skew 500 ppm
I20260620 03:10:21.731755 18066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:10:21.738524 18066 webserver.cc:460] Webserver started at http://127.0.97.62:46231/ using document root <none> and password file <none>
I20260620 03:10:21.739156 18066 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:21.739240 18066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:21.739609 18066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:21.742379 18066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "e6ea139bcfe644c19b17c78ad517dd9f"
format_stamp: "Formatted at 2026-06-20 03:10:21 on dist-test-slave-xv68"
server_key: "62202304ff856241ce275b27f55e3efc"
server_key_iv: "37d549ecca32939880a2f5b1f4c0ceed"
server_key_version: "encryptionkey@0"
I20260620 03:10:21.743109 18066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "e6ea139bcfe644c19b17c78ad517dd9f"
format_stamp: "Formatted at 2026-06-20 03:10:21 on dist-test-slave-xv68"
server_key: "62202304ff856241ce275b27f55e3efc"
server_key_iv: "37d549ecca32939880a2f5b1f4c0ceed"
server_key_version: "encryptionkey@0"
I20260620 03:10:21.748085 18066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 03:10:21.751430 18080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:21.752781 18066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:10:21.753011 18066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "e6ea139bcfe644c19b17c78ad517dd9f"
format_stamp: "Formatted at 2026-06-20 03:10:21 on dist-test-slave-xv68"
server_key: "62202304ff856241ce275b27f55e3efc"
server_key_iv: "37d549ecca32939880a2f5b1f4c0ceed"
server_key_version: "encryptionkey@0"
I20260620 03:10:21.753252 18066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:21.779544 18066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:21.780354 18066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:21.780654 18066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:21.798157 18066 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40749
I20260620 03:10:21.798182 18131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40749 every 8 connection(s)
I20260620 03:10:21.799476 18066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:10:21.801462 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 18066
I20260620 03:10:21.801669 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:10:21.802111 388 external_mini_cluster.cc:1428] Setting key 480a092ed5af486be40d710ddf7414d6
I20260620 03:10:21.803350 18132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:21.816902 18135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:21.803839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47440 (local address 127.0.97.62:40749)
0620 03:10:21.804891 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:21.804913 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:21.804962 (+ 49us) server_negotiation.cc:407] Connection header received
0620 03:10:21.806751 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:21.806778 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:21.807404 (+ 626us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:21.807813 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:21.808323 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:21.810418 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:21.811835 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:21.812825 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:21.813004 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:21.814152 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:21.814163 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:21.814180 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:21.814930 (+ 750us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:21.814998 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:21.815023 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:21.815161 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:21.815377 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 03:10:21.815713 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":386,"threads_started":1}
I20260620 03:10:21.817996 18132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f: Bootstrap starting.
I20260620 03:10:21.821244 18132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:21.822122 18132 log.cc:824] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:21.824913 18132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f: No bootstrap required, opened a new log
I20260620 03:10:21.832612 18132 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ea139bcfe644c19b17c78ad517dd9f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40749 } }
I20260620 03:10:21.832963 18132 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:21.833058 18132 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ea139bcfe644c19b17c78ad517dd9f, State: Initialized, Role: FOLLOWER
I20260620 03:10:21.833626 18132 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ea139bcfe644c19b17c78ad517dd9f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40749 } }
I20260620 03:10:21.833858 18132 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:10:21.833979 18132 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:10:21.834143 18132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:21.836663 18132 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ea139bcfe644c19b17c78ad517dd9f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40749 } }
I20260620 03:10:21.837139 18132 leader_election.cc:302] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6ea139bcfe644c19b17c78ad517dd9f; no voters:
I20260620 03:10:21.837736 18132 leader_election.cc:288] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:10:21.837910 18137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:21.838701 18137 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f [term 1 LEADER]: Becoming Leader. State: Replica: e6ea139bcfe644c19b17c78ad517dd9f, State: Running, Role: LEADER
I20260620 03:10:21.839193 18137 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ea139bcfe644c19b17c78ad517dd9f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40749 } }
I20260620 03:10:21.839869 18132 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:10:21.841835 18138 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6ea139bcfe644c19b17c78ad517dd9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ea139bcfe644c19b17c78ad517dd9f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40749 } } }
I20260620 03:10:21.842444 18138 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:21.842339 18139 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6ea139bcfe644c19b17c78ad517dd9f. Latest consensus state: current_term: 1 leader_uuid: "e6ea139bcfe644c19b17c78ad517dd9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ea139bcfe644c19b17c78ad517dd9f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40749 } } }
I20260620 03:10:21.847748 18139 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:21.848608 18145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:10:21.854553 18145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:10:21.863104 18145 catalog_manager.cc:1347] Generated new cluster ID: 9967695194274dcaa1bc3423564c4e44
I20260620 03:10:21.863233 18145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:10:21.885726 18145 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:10:21.886989 18145 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:10:21.899317 18145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f: Generated new TSK 0
I20260620 03:10:21.900208 18145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:10:21.905504 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40749
--builtin_ntp_servers=127.0.97.20:41815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 03:10:22.246575 18156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:22.246851 18156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:22.246997 18156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:22.254709 18156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:22.254799 18156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:22.254998 18156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:10:22.264277 18156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:22.265503 18156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:22.267220 18156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:22.279865 18161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:22.279894 18162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:22.279932 18164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:22.281247 18156 server_base.cc:1034] running on GCE node
I20260620 03:10:22.281867 18156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:22.282780 18156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:22.283968 18156 hybrid_clock.cc:648] HybridClock initialized: now 1781925022283913 us; error 55 us; skew 500 ppm
I20260620 03:10:22.284271 18156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:10:22.286309 18156 webserver.cc:460] Webserver started at http://127.0.97.1:46555/ using document root <none> and password file <none>
I20260620 03:10:22.286933 18156 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:22.287047 18156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:22.287443 18156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:22.290227 18156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "ca2ff2d9376e435092252cd06990be75"
format_stamp: "Formatted at 2026-06-20 03:10:22 on dist-test-slave-xv68"
server_key: "36bbe8d15e2f1c907ba5d40d4f370fd5"
server_key_iv: "3d5c0c3e597b603e1f451ebf15ca6d6b"
server_key_version: "encryptionkey@0"
I20260620 03:10:22.290980 18156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ca2ff2d9376e435092252cd06990be75"
format_stamp: "Formatted at 2026-06-20 03:10:22 on dist-test-slave-xv68"
server_key: "36bbe8d15e2f1c907ba5d40d4f370fd5"
server_key_iv: "3d5c0c3e597b603e1f451ebf15ca6d6b"
server_key_version: "encryptionkey@0"
I20260620 03:10:22.295828 18156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 03:10:22.299043 18170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:22.300340 18156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:10:22.300505 18156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "ca2ff2d9376e435092252cd06990be75"
format_stamp: "Formatted at 2026-06-20 03:10:22 on dist-test-slave-xv68"
server_key: "36bbe8d15e2f1c907ba5d40d4f370fd5"
server_key_iv: "3d5c0c3e597b603e1f451ebf15ca6d6b"
server_key_version: "encryptionkey@0"
I20260620 03:10:22.300936 18156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:22.325961 18156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:22.326807 18156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:22.327086 18156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:22.328099 18156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:22.329578 18156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:22.329699 18156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:22.329840 18156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:22.329931 18156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:22.363740 18156 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42135
I20260620 03:10:22.363790 18282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42135 every 8 connection(s)
I20260620 03:10:22.365068 18156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:10:22.373598 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 18156
I20260620 03:10:22.373759 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:10:22.374224 388 external_mini_cluster.cc:1428] Setting key 1c91c2fb740536ba518ffe27651d25ff
I20260620 03:10:22.377311 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40749
--builtin_ntp_servers=127.0.97.20:41815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 03:10:22.377993 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:22.367730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48757 (local address 127.0.97.62:40749)
0620 03:10:22.368141 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:22.368145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:22.369351 (+ 1206us) server_negotiation.cc:407] Connection header received
0620 03:10:22.371098 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:22.371102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:22.371179 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:22.371275 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:22.372315 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:22.373016 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:22.374697 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:22.375265 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:22.375421 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:22.376983 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:22.376992 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:22.377000 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:22.377024 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:22.377078 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:22.377097 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:22.377198 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:22.377397 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 03:10:22.377521 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":198,"threads_started":1}
I20260620 03:10:22.378865 18286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:22.368291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40749 (local address 127.0.97.1:48757)
0620 03:10:22.369200 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:22.369241 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:22.370863 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:22.371533 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:22.371541 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:22.371641 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:10:22.372168 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:22.372183 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:22.373141 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:22.373145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:22.374534 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:22.374545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:22.375522 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:22.375530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:22.375840 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:22.376664 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:10:22.376666 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:10:22.376710 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:10:22.376782 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:10:22.377199 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:10:22.377207 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:10:22.377213 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 03:10:22.377357 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 03:10:22.377768 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":310,"threads_started":1}
I20260620 03:10:22.380398 18283 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40749
I20260620 03:10:22.380719 18283 heartbeater.cc:464] Registering TS with master...
I20260620 03:10:22.381560 18283 heartbeater.cc:511] Master 127.0.97.62:40749 requested a full tablet report, sending...
I20260620 03:10:22.383803 18097 ts_manager.cc:194] Registered new tserver with Master: ca2ff2d9376e435092252cd06990be75 (127.0.97.1:42135)
I20260620 03:10:22.385341 18097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:48757
W20260620 03:10:22.701001 18287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:22.701352 18287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:22.701512 18287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:22.708714 18287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:22.708804 18287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:22.709009 18287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:10:22.717916 18287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:22.719197 18287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:22.720973 18287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:22.733556 18292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:22.733632 18295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:22.733649 18293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:22.734452 18287 server_base.cc:1034] running on GCE node
I20260620 03:10:22.735001 18287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:22.735934 18287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:22.737164 18287 hybrid_clock.cc:648] HybridClock initialized: now 1781925022737108 us; error 72 us; skew 500 ppm
I20260620 03:10:22.737483 18287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:10:22.739638 18287 webserver.cc:460] Webserver started at http://127.0.97.2:33923/ using document root <none> and password file <none>
I20260620 03:10:22.740310 18287 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:22.740442 18287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:22.740756 18287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:22.743536 18287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "ffccb7cc5a3f46aca18d81f4a97c9edf"
format_stamp: "Formatted at 2026-06-20 03:10:22 on dist-test-slave-xv68"
server_key: "64f650871f5238c613e9b265700aa304"
server_key_iv: "997d6008e6197d261294167af0b0b7e1"
server_key_version: "encryptionkey@0"
I20260620 03:10:22.744306 18287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ffccb7cc5a3f46aca18d81f4a97c9edf"
format_stamp: "Formatted at 2026-06-20 03:10:22 on dist-test-slave-xv68"
server_key: "64f650871f5238c613e9b265700aa304"
server_key_iv: "997d6008e6197d261294167af0b0b7e1"
server_key_version: "encryptionkey@0"
I20260620 03:10:22.749048 18287 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 03:10:22.752396 18301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:22.753688 18287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:10:22.753861 18287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "ffccb7cc5a3f46aca18d81f4a97c9edf"
format_stamp: "Formatted at 2026-06-20 03:10:22 on dist-test-slave-xv68"
server_key: "64f650871f5238c613e9b265700aa304"
server_key_iv: "997d6008e6197d261294167af0b0b7e1"
server_key_version: "encryptionkey@0"
I20260620 03:10:22.754069 18287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:22.783022 18287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:22.783872 18287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:22.784157 18287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:22.785087 18287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:22.786511 18287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:22.786635 18287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:22.786778 18287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:22.786870 18287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:22.826031 18287 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36297
I20260620 03:10:22.826126 18413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36297 every 8 connection(s)
I20260620 03:10:22.827353 18287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:10:22.832196 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 18287
I20260620 03:10:22.832362 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:10:22.832803 388 external_mini_cluster.cc:1428] Setting key 4edc7aad357812ec39c3984f5a20892e
I20260620 03:10:22.835549 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40749
--builtin_ntp_servers=127.0.97.20:41815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 03:10:22.839689 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:22.829821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60395 (local address 127.0.97.62:40749)
0620 03:10:22.829999 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:22.830004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:22.831323 (+ 1319us) server_negotiation.cc:407] Connection header received
0620 03:10:22.833248 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:22.833254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:22.833344 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:22.833455 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:22.834699 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:22.835471 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:22.837019 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:22.837411 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:22.837552 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:22.838863 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:22.838868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:22.838871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:22.838887 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:22.838925 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:22.838936 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:22.839031 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:22.839244 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 03:10:22.839364 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 03:10:22.840766 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:22.830293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40749 (local address 127.0.97.2:60395)
0620 03:10:22.831136 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:22.831176 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:22.832993 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:22.833708 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:22.833719 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:22.833826 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:10:22.834540 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:22.834558 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:22.835628 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:22.835633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:22.836876 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:22.836887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:22.837657 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:22.837667 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:22.837872 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:22.838542 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:10:22.838545 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:10:22.838581 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:10:22.838666 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:10:22.839031 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:10:22.839044 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:10:22.839053 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 03:10:22.839232 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 03:10:22.839669 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":272,"threads_started":1}
I20260620 03:10:22.842139 18414 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40749
I20260620 03:10:22.842430 18414 heartbeater.cc:464] Registering TS with master...
I20260620 03:10:22.843089 18414 heartbeater.cc:511] Master 127.0.97.62:40749 requested a full tablet report, sending...
I20260620 03:10:22.844456 18097 ts_manager.cc:194] Registered new tserver with Master: ffccb7cc5a3f46aca18d81f4a97c9edf (127.0.97.2:36297)
I20260620 03:10:22.845259 18097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:60395
W20260620 03:10:23.159991 18417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:23.160301 18417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:23.160457 18417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:23.167549 18417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:23.167676 18417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:23.167860 18417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:10:23.176882 18417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:23.178148 18417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:23.179895 18417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:23.192803 18423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:23.192826 18425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:23.192759 18422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:23.193571 18417 server_base.cc:1034] running on GCE node
I20260620 03:10:23.194160 18417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:23.195116 18417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:23.196352 18417 hybrid_clock.cc:648] HybridClock initialized: now 1781925023196313 us; error 88 us; skew 500 ppm
I20260620 03:10:23.196686 18417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:10:23.198848 18417 webserver.cc:460] Webserver started at http://127.0.97.3:36163/ using document root <none> and password file <none>
I20260620 03:10:23.199587 18417 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:23.199712 18417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:23.200038 18417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:23.202777 18417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "dd4b0c4dbbbe4b0b81708ca417299f63"
format_stamp: "Formatted at 2026-06-20 03:10:23 on dist-test-slave-xv68"
server_key: "13b358ac30695da2a70a12a718201cd3"
server_key_iv: "6a45c6323e78cd8c2bae417db35e1770"
server_key_version: "encryptionkey@0"
I20260620 03:10:23.203624 18417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dd4b0c4dbbbe4b0b81708ca417299f63"
format_stamp: "Formatted at 2026-06-20 03:10:23 on dist-test-slave-xv68"
server_key: "13b358ac30695da2a70a12a718201cd3"
server_key_iv: "6a45c6323e78cd8c2bae417db35e1770"
server_key_version: "encryptionkey@0"
I20260620 03:10:23.208741 18417 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 03:10:23.212218 18431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:23.213591 18417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:10:23.213766 18417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "dd4b0c4dbbbe4b0b81708ca417299f63"
format_stamp: "Formatted at 2026-06-20 03:10:23 on dist-test-slave-xv68"
server_key: "13b358ac30695da2a70a12a718201cd3"
server_key_iv: "6a45c6323e78cd8c2bae417db35e1770"
server_key_version: "encryptionkey@0"
I20260620 03:10:23.213999 18417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:23.249323 18417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:23.250181 18417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:23.250463 18417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:23.251506 18417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:23.252940 18417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:23.253069 18417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:23.253206 18417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:23.253299 18417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:23.286547 18417 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46631
I20260620 03:10:23.286635 18543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46631 every 8 connection(s)
I20260620 03:10:23.287930 18417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:10:23.289458 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 18417
I20260620 03:10:23.289644 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:10:23.290097 388 external_mini_cluster.cc:1428] Setting key 399972861a4377888d20388d320a36f9
I20260620 03:10:23.300516 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.290675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57271 (local address 127.0.97.62:40749)
0620 03:10:23.290878 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:23.290884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:23.292564 (+ 1680us) server_negotiation.cc:407] Connection header received
0620 03:10:23.294337 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:23.294341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:23.294411 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:23.294508 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:23.295641 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.296371 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.297897 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.298313 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.298406 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:23.299777 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:23.299783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:23.299786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:23.299803 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:23.299835 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:23.299847 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:23.299936 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:23.300205 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 03:10:23.300281 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:10:23.301460 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.291215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40749 (local address 127.0.97.3:57271)
0620 03:10:23.292373 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:23.292422 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:23.294089 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:23.294721 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:23.294729 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:23.294807 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:10:23.295437 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:23.295504 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.296538 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.296542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:23.297766 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:23.297777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.298535 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.298540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:23.298725 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:23.299409 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:10:23.299411 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:10:23.299492 (+ 81us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:10:23.299572 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:10:23.299972 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:10:23.299981 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:10:23.299987 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 03:10:23.300137 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 03:10:23.300407 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":417,"threads_started":1}
I20260620 03:10:23.302721 18544 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40749
I20260620 03:10:23.303017 18544 heartbeater.cc:464] Registering TS with master...
I20260620 03:10:23.303893 18544 heartbeater.cc:511] Master 127.0.97.62:40749 requested a full tablet report, sending...
I20260620 03:10:23.305423 18097 ts_manager.cc:194] Registered new tserver with Master: dd4b0c4dbbbe4b0b81708ca417299f63 (127.0.97.3:46631)
I20260620 03:10:23.306402 18097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:57271
I20260620 03:10:23.317677 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:10:23.327746 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.321994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47442 (local address 127.0.97.62:40749)
0620 03:10:23.322226 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:23.322236 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:23.322644 (+ 408us) server_negotiation.cc:407] Connection header received
0620 03:10:23.322791 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:23.322795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:23.322870 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:23.322955 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:23.323473 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.324228 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.325568 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.325962 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.326064 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:23.327005 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:23.327009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:23.327012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:23.327029 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:23.327061 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:23.327073 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:23.327141 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:23.327257 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 03:10:23.327396 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 03:10:23.331355 18097 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:10:23.338042 18097 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:10:23.365334 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.349854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46631 (local address 127.0.0.1:38212)
0620 03:10:23.351984 (+ 2130us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:23.352025 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:23.352254 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:23.353567 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:23.353577 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:23.353622 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:23.353984 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:23.354000 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.358909 (+ 4909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.358918 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:23.362076 (+ 3158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:23.362096 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.364360 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.364369 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:23.364636 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:23.364705 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 03:10:23.364820 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 03:10:23.365010 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1870,"thread_start_us":12836,"threads_started":1}
I20260620 03:10:23.366068 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.350491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.0.97.3:46631)
0620 03:10:23.351552 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:23.351564 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:23.352149 (+ 585us) server_negotiation.cc:407] Connection header received
0620 03:10:23.352735 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:23.352746 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:23.353033 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:23.353233 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:23.354293 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.358697 (+ 4404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.362313 (+ 3616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.363796 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.363894 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:23.364181 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:23.365555 (+ 1374us) server_negotiation.cc:299] Negotiation successful
0620 03:10:23.365719 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":2632,"threads_started":1}
I20260620 03:10:23.369776 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.350560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36297 (local address 127.0.0.1:53754)
0620 03:10:23.363011 (+ 12451us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:23.363030 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:23.363147 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:23.364132 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:23.364140 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:23.364169 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:23.364591 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:23.364602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.366408 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.366413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:23.367825 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:23.367837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.369062 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.369068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:23.369239 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:23.369271 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 03:10:23.369362 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 03:10:23.369498 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12308,"thread_start_us":983,"threads_started":1}
I20260620 03:10:23.370612 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.351260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53754 (local address 127.0.97.2:36297)
0620 03:10:23.362400 (+ 11140us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:23.362409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:23.363268 (+ 859us) server_negotiation.cc:407] Connection header received
0620 03:10:23.363395 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:23.363403 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:23.363722 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:23.363914 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:23.364767 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.366252 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.367999 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.368872 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.369026 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:23.369263 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:23.370129 (+ 866us) server_negotiation.cc:299] Negotiation successful
0620 03:10:23.370245 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10953,"thread_start_us":11118,"threads_started":1}
I20260620 03:10:23.373543 18479 tablet_service.cc:1467] Processing CreateTablet for tablet ddba503d8cdf451c94f9507f15e47737 (DEFAULT_TABLE table=test-table [id=a1744237a80a4263900e8d988fac3802]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:10:23.373862 18349 tablet_service.cc:1467] Processing CreateTablet for tablet ddba503d8cdf451c94f9507f15e47737 (DEFAULT_TABLE table=test-table [id=a1744237a80a4263900e8d988fac3802]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:10:23.375136 18349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddba503d8cdf451c94f9507f15e47737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:23.375135 18479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddba503d8cdf451c94f9507f15e47737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:23.382525 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.369275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42135 (local address 127.0.0.1:41478)
0620 03:10:23.369439 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:23.369456 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:23.369534 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:23.377412 (+ 7878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:23.377417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:23.377444 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:23.377629 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:23.377643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.379001 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.379006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:23.380561 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:23.380576 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.381778 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.381784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:23.381954 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:23.381995 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 03:10:23.382105 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 03:10:23.382249 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 03:10:23.382645 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.363364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.0.97.1:42135)
0620 03:10:23.376986 (+ 13622us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:23.376992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:23.377013 (+ 21us) server_negotiation.cc:407] Connection header received
0620 03:10:23.377084 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:23.377089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:23.377235 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:23.377434 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:23.377813 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.378854 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.380761 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.381608 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.381768 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:23.382022 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:23.382145 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 03:10:23.382296 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13485,"thread_start_us":14259,"threads_started":1}
I20260620 03:10:23.385756 18218 tablet_service.cc:1467] Processing CreateTablet for tablet ddba503d8cdf451c94f9507f15e47737 (DEFAULT_TABLE table=test-table [id=a1744237a80a4263900e8d988fac3802]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:10:23.387152 18218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddba503d8cdf451c94f9507f15e47737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:23.388175 18283 heartbeater.cc:503] Master 127.0.97.62:40749 was elected leader, sending a full tablet report...
I20260620 03:10:23.391634 18562 tablet_bootstrap.cc:492] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63: Bootstrap starting.
I20260620 03:10:23.395985 18561 tablet_bootstrap.cc:492] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf: Bootstrap starting.
I20260620 03:10:23.395279 18562 tablet_bootstrap.cc:654] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:23.398834 18562 log.cc:824] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:23.400861 18561 tablet_bootstrap.cc:654] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:23.402343 18561 log.cc:824] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:23.406039 18562 tablet_bootstrap.cc:492] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63: No bootstrap required, opened a new log
I20260620 03:10:23.406320 18562 ts_tablet_manager.cc:1397] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.003s
I20260620 03:10:23.412009 18564 tablet_bootstrap.cc:492] T ddba503d8cdf451c94f9507f15e47737 P ca2ff2d9376e435092252cd06990be75: Bootstrap starting.
I20260620 03:10:23.415454 18564 tablet_bootstrap.cc:654] T ddba503d8cdf451c94f9507f15e47737 P ca2ff2d9376e435092252cd06990be75: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:23.416647 18564 log.cc:824] T ddba503d8cdf451c94f9507f15e47737 P ca2ff2d9376e435092252cd06990be75: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:23.420621 18561 tablet_bootstrap.cc:492] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf: No bootstrap required, opened a new log
I20260620 03:10:23.420930 18561 ts_tablet_manager.cc:1397] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.015s
I20260620 03:10:23.423076 18562 raft_consensus.cc:348] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4b0c4dbbbe4b0b81708ca417299f63" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46631 } } peers { permanent_uuid: "ca2ff2d9376e435092252cd06990be75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42135 } } peers { permanent_uuid: "ffccb7cc5a3f46aca18d81f4a97c9edf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36297 } }
I20260620 03:10:23.423691 18562 raft_consensus.cc:374] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:23.423832 18562 raft_consensus.cc:729] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4b0c4dbbbe4b0b81708ca417299f63, State: Initialized, Role: FOLLOWER
I20260620 03:10:23.424707 18562 consensus_queue.cc:260] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4b0c4dbbbe4b0b81708ca417299f63" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46631 } } peers { permanent_uuid: "ca2ff2d9376e435092252cd06990be75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42135 } } peers { permanent_uuid: "ffccb7cc5a3f46aca18d81f4a97c9edf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36297 } }
I20260620 03:10:23.424961 18564 tablet_bootstrap.cc:492] T ddba503d8cdf451c94f9507f15e47737 P ca2ff2d9376e435092252cd06990be75: No bootstrap required, opened a new log
I20260620 03:10:23.425272 18564 ts_tablet_manager.cc:1397] T ddba503d8cdf451c94f9507f15e47737 P ca2ff2d9376e435092252cd06990be75: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.008s
I20260620 03:10:23.425923 18544 heartbeater.cc:503] Master 127.0.97.62:40749 was elected leader, sending a full tablet report...
I20260620 03:10:23.426482 18562 ts_tablet_manager.cc:1428] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63: Time spent starting tablet: real 0.020s user 0.014s sys 0.000s
I20260620 03:10:23.431017 18561 raft_consensus.cc:348] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4b0c4dbbbe4b0b81708ca417299f63" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46631 } } peers { permanent_uuid: "ca2ff2d9376e435092252cd06990be75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42135 } } peers { permanent_uuid: "ffccb7cc5a3f46aca18d81f4a97c9edf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36297 } }
I20260620 03:10:23.431478 18561 raft_consensus.cc:374] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:23.431593 18561 raft_consensus.cc:729] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffccb7cc5a3f46aca18d81f4a97c9edf, State: Initialized, Role: FOLLOWER
I20260620 03:10:23.432221 18561 consensus_queue.cc:260] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4b0c4dbbbe4b0b81708ca417299f63" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46631 } } peers { permanent_uuid: "ca2ff2d9376e435092252cd06990be75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42135 } } peers { permanent_uuid: "ffccb7cc5a3f46aca18d81f4a97c9edf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36297 } }
I20260620 03:10:23.433341 18414 heartbeater.cc:503] Master 127.0.97.62:40749 was elected leader, sending a full tablet report...
I20260620 03:10:23.433723 18561 ts_tablet_manager.cc:1428] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260620 03:10:23.435513 18564 raft_consensus.cc:348] T ddba503d8cdf451c94f9507f15e47737 P ca2ff2d9376e435092252cd06990be75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4b0c4dbbbe4b0b81708ca417299f63" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46631 } } peers { permanent_uuid: "ca2ff2d9376e435092252cd06990be75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42135 } } peers { permanent_uuid: "ffccb7cc5a3f46aca18d81f4a97c9edf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36297 } }
I20260620 03:10:23.435890 18564 raft_consensus.cc:374] T ddba503d8cdf451c94f9507f15e47737 P ca2ff2d9376e435092252cd06990be75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:23.436004 18564 raft_consensus.cc:729] T ddba503d8cdf451c94f9507f15e47737 P ca2ff2d9376e435092252cd06990be75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2ff2d9376e435092252cd06990be75, State: Initialized, Role: FOLLOWER
I20260620 03:10:23.436575 18564 consensus_queue.cc:260] T ddba503d8cdf451c94f9507f15e47737 P ca2ff2d9376e435092252cd06990be75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4b0c4dbbbe4b0b81708ca417299f63" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46631 } } peers { permanent_uuid: "ca2ff2d9376e435092252cd06990be75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42135 } } peers { permanent_uuid: "ffccb7cc5a3f46aca18d81f4a97c9edf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36297 } }
I20260620 03:10:23.438052 18564 ts_tablet_manager.cc:1428] T ddba503d8cdf451c94f9507f15e47737 P ca2ff2d9376e435092252cd06990be75: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260620 03:10:23.440925 18567 raft_consensus.cc:482] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:10:23.441201 18567 raft_consensus.cc:504] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4b0c4dbbbe4b0b81708ca417299f63" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46631 } } peers { permanent_uuid: "ca2ff2d9376e435092252cd06990be75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42135 } } peers { permanent_uuid: "ffccb7cc5a3f46aca18d81f4a97c9edf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36297 } }
I20260620 03:10:23.442657 18567 leader_election.cc:288] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca2ff2d9376e435092252cd06990be75 (127.0.97.1:42135), ffccb7cc5a3f46aca18d81f4a97c9edf (127.0.97.2:36297)
I20260620 03:10:23.450003 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.443348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51759 (local address 127.0.97.1:42135)
0620 03:10:23.443537 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:23.443543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:23.443561 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:10:23.443621 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:23.443626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:23.443703 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:23.443790 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:23.444271 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.445232 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.447753 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.449043 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.449156 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:23.449252 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:23.449653 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 03:10:23.449773 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 03:10:23.450019 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.443186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38195 (local address 127.0.97.2:36297)
0620 03:10:23.443386 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:23.443393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:23.443907 (+ 514us) server_negotiation.cc:407] Connection header received
0620 03:10:23.444034 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:23.444038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:23.444090 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:23.444157 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:23.444676 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.445577 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.448013 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.449158 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.449276 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:23.449369 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:23.449648 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 03:10:23.449770 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 03:10:23.450053 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.442872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42135 (local address 127.0.97.3:51759)
0620 03:10:23.443259 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:23.443279 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:23.443461 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:23.443861 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:23.443868 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:23.443899 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:23.444096 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:23.444104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.445360 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.445363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:23.447548 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:23.447562 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.449211 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.449220 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:23.449429 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:23.449494 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 03:10:23.449619 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 03:10:23.449767 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":128}
I20260620 03:10:23.450098 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.442872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36297 (local address 127.0.97.3:38195)
0620 03:10:23.443795 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:23.443812 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:23.443922 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:23.444206 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:23.444214 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:23.444240 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:23.444538 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:23.444545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.445711 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.445716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:23.447778 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:23.447791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.449326 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.449331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:23.449490 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:23.449516 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 03:10:23.449612 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 03:10:23.449741 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":62,"thread_start_us":637,"threads_started":1}
I20260620 03:10:23.451392 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddba503d8cdf451c94f9507f15e47737" candidate_uuid: "dd4b0c4dbbbe4b0b81708ca417299f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca2ff2d9376e435092252cd06990be75" is_pre_election: true
I20260620 03:10:23.451493 18369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddba503d8cdf451c94f9507f15e47737" candidate_uuid: "dd4b0c4dbbbe4b0b81708ca417299f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffccb7cc5a3f46aca18d81f4a97c9edf" is_pre_election: true
I20260620 03:10:23.451892 18238 raft_consensus.cc:2454] T ddba503d8cdf451c94f9507f15e47737 P ca2ff2d9376e435092252cd06990be75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd4b0c4dbbbe4b0b81708ca417299f63 in term 0.
I20260620 03:10:23.452000 18369 raft_consensus.cc:2454] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd4b0c4dbbbe4b0b81708ca417299f63 in term 0.
I20260620 03:10:23.452589 18432 leader_election.cc:302] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca2ff2d9376e435092252cd06990be75, dd4b0c4dbbbe4b0b81708ca417299f63; no voters:
I20260620 03:10:23.452972 18567 raft_consensus.cc:2781] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:10:23.453135 18567 raft_consensus.cc:482] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:10:23.453251 18567 raft_consensus.cc:3037] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:23.455438 18567 raft_consensus.cc:504] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4b0c4dbbbe4b0b81708ca417299f63" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46631 } } peers { permanent_uuid: "ca2ff2d9376e435092252cd06990be75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42135 } } peers { permanent_uuid: "ffccb7cc5a3f46aca18d81f4a97c9edf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36297 } }
I20260620 03:10:23.456032 18567 leader_election.cc:288] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [CANDIDATE]: Term 1 election: Requested vote from peers ca2ff2d9376e435092252cd06990be75 (127.0.97.1:42135), ffccb7cc5a3f46aca18d81f4a97c9edf (127.0.97.2:36297)
I20260620 03:10:23.456506 18369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddba503d8cdf451c94f9507f15e47737" candidate_uuid: "dd4b0c4dbbbe4b0b81708ca417299f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffccb7cc5a3f46aca18d81f4a97c9edf"
I20260620 03:10:23.456619 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddba503d8cdf451c94f9507f15e47737" candidate_uuid: "dd4b0c4dbbbe4b0b81708ca417299f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca2ff2d9376e435092252cd06990be75"
I20260620 03:10:23.456743 18369 raft_consensus.cc:3037] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:23.456786 18238 raft_consensus.cc:3037] T ddba503d8cdf451c94f9507f15e47737 P ca2ff2d9376e435092252cd06990be75 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:23.459959 18238 raft_consensus.cc:2454] T ddba503d8cdf451c94f9507f15e47737 P ca2ff2d9376e435092252cd06990be75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd4b0c4dbbbe4b0b81708ca417299f63 in term 1.
I20260620 03:10:23.459975 18369 raft_consensus.cc:2454] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd4b0c4dbbbe4b0b81708ca417299f63 in term 1.
I20260620 03:10:23.460314 18432 leader_election.cc:302] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca2ff2d9376e435092252cd06990be75, dd4b0c4dbbbe4b0b81708ca417299f63; no voters:
I20260620 03:10:23.460606 18567 raft_consensus.cc:2781] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:23.461058 18567 raft_consensus.cc:686] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [term 1 LEADER]: Becoming Leader. State: Replica: dd4b0c4dbbbe4b0b81708ca417299f63, State: Running, Role: LEADER
I20260620 03:10:23.461498 18567 consensus_queue.cc:237] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4b0c4dbbbe4b0b81708ca417299f63" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46631 } } peers { permanent_uuid: "ca2ff2d9376e435092252cd06990be75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42135 } } peers { permanent_uuid: "ffccb7cc5a3f46aca18d81f4a97c9edf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36297 } }
I20260620 03:10:23.466120 18097 catalog_manager.cc:5495] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd4b0c4dbbbe4b0b81708ca417299f63 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "dd4b0c4dbbbe4b0b81708ca417299f63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4b0c4dbbbe4b0b81708ca417299f63" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca2ff2d9376e435092252cd06990be75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffccb7cc5a3f46aca18d81f4a97c9edf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36297 } health_report { overall_health: UNKNOWN } } }
I20260620 03:10:23.483054 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.478375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.0.97.3:46631)
0620 03:10:23.478536 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:23.478540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:23.478556 (+ 16us) server_negotiation.cc:407] Connection header received
0620 03:10:23.478704 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:23.478707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:23.478778 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:23.478895 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:10:23.479201 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.479966 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.481288 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.481677 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.481764 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:23.482172 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:10:23.482610 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:10:23.482719 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:23.482822 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 03:10:23.482880 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 03:10:23.490861 18238 raft_consensus.cc:1261] T ddba503d8cdf451c94f9507f15e47737 P ca2ff2d9376e435092252cd06990be75 [term 1 FOLLOWER]: Refusing update from remote peer dd4b0c4dbbbe4b0b81708ca417299f63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:10:23.490861 18369 raft_consensus.cc:1261] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf [term 1 FOLLOWER]: Refusing update from remote peer dd4b0c4dbbbe4b0b81708ca417299f63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:10:23.491906 18571 consensus_queue.cc:1035] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca2ff2d9376e435092252cd06990be75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:10:23.492244 18567 consensus_queue.cc:1035] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffccb7cc5a3f46aca18d81f4a97c9edf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:10:23.505326 18575 mvcc.cc:205] Tried to move back new op lower bound from 7298764896208691200 to 7298764896102113280. Current Snapshot: MvccSnapshot[applied={T|T < 7298764896208691200}]
I20260620 03:10:23.510133 18576 mvcc.cc:205] Tried to move back new op lower bound from 7298764896208691200 to 7298764896102113280. Current Snapshot: MvccSnapshot[applied={T|T < 7298764896208691200 or (T in {7298764896208691200})}]
I20260620 03:10:23.515832 18096 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:10:23.516105 18096 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:10:23.520097 18096 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6ea139bcfe644c19b17c78ad517dd9f: Sending DeleteTablet for 3 replicas of tablet ddba503d8cdf451c94f9507f15e47737
I20260620 03:10:23.521322 18479 tablet_service.cc:1514] Processing DeleteTablet for tablet ddba503d8cdf451c94f9507f15e47737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:10:23 UTC) from {username='slave'} at 127.0.0.1:38212
I20260620 03:10:23.521314 18218 tablet_service.cc:1514] Processing DeleteTablet for tablet ddba503d8cdf451c94f9507f15e47737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:10:23 UTC) from {username='slave'} at 127.0.0.1:41478
I20260620 03:10:23.521328 18349 tablet_service.cc:1514] Processing DeleteTablet for tablet ddba503d8cdf451c94f9507f15e47737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:10:23 UTC) from {username='slave'} at 127.0.0.1:53754
I20260620 03:10:23.525341 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 18156
I20260620 03:10:23.541822 18584 tablet_replica.cc:330] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf: stopping tablet replica
I20260620 03:10:23.542035 18585 tablet_replica.cc:330] T ddba503d8cdf451c94f9507f15e47737 P ca2ff2d9376e435092252cd06990be75: stopping tablet replica
I20260620 03:10:23.542320 18584 raft_consensus.cc:2229] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:10:23.542423 18585 raft_consensus.cc:2229] T ddba503d8cdf451c94f9507f15e47737 P ca2ff2d9376e435092252cd06990be75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:10:23.542865 18585 raft_consensus.cc:2258] T ddba503d8cdf451c94f9507f15e47737 P ca2ff2d9376e435092252cd06990be75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:10:23.542865 18584 raft_consensus.cc:2258] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:10:23.543702 18583 tablet_replica.cc:330] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63: stopping tablet replica
I20260620 03:10:23.544095 18583 raft_consensus.cc:2229] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:10:23.544795 18583 raft_consensus.cc:2258] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:10:23.545284 18584 ts_tablet_manager.cc:1905] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:10:23.546835 18583 ts_tablet_manager.cc:1905] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:10:23.547149 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.526379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41494 (local address 127.0.97.1:42135)
0620 03:10:23.526615 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:23.526623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:23.529949 (+ 3326us) server_negotiation.cc:407] Connection header received
0620 03:10:23.530837 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:23.530847 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:23.531019 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:23.531147 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:23.532457 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.533995 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.539028 (+ 5033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.539859 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.539967 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:23.545497 (+ 5530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:23.545519 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:23.545535 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:23.546240 (+ 705us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:23.546306 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:23.546329 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:23.546424 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:23.546673 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 03:10:23.546809 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 03:10:23.548290 18278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:23.553634 18584 ts_tablet_manager.cc:1918] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:10:23.553859 18584 log.cc:1196] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/ddba503d8cdf451c94f9507f15e47737
I20260620 03:10:23.554510 18584 ts_tablet_manager.cc:1939] T ddba503d8cdf451c94f9507f15e47737 P ffccb7cc5a3f46aca18d81f4a97c9edf: Deleting consensus metadata
I20260620 03:10:23.554562 18583 ts_tablet_manager.cc:1918] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:10:23.554865 18583 log.cc:1196] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/ddba503d8cdf451c94f9507f15e47737
I20260620 03:10:23.555521 18583 ts_tablet_manager.cc:1939] T ddba503d8cdf451c94f9507f15e47737 P dd4b0c4dbbbe4b0b81708ca417299f63: Deleting consensus metadata
I20260620 03:10:23.556991 18084 catalog_manager.cc:4841] TS ffccb7cc5a3f46aca18d81f4a97c9edf (127.0.97.2:36297): tablet ddba503d8cdf451c94f9507f15e47737 (table test-table [id=a1744237a80a4263900e8d988fac3802]) successfully deleted
I20260620 03:10:23.557515 18081 catalog_manager.cc:4841] TS dd4b0c4dbbbe4b0b81708ca417299f63 (127.0.97.3:46631): tablet ddba503d8cdf451c94f9507f15e47737 (table test-table [id=a1744237a80a4263900e8d988fac3802]) successfully deleted
I20260620 03:10:23.548949 18585 ts_tablet_manager.cc:1905] T ddba503d8cdf451c94f9507f15e47737 P ca2ff2d9376e435092252cd06990be75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:10:23.679024 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 18156
W20260620 03:10:23.703080 18081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42135 (error 108)
I20260620 03:10:23.703773 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 18287
I20260620 03:10:23.705354 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.704912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42135 (local address 127.0.0.1:41498)
0620 03:10:23.705110 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:23.705212 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 03:10:23.705610 18081 catalog_manager.cc:4584] TS ca2ff2d9376e435092252cd06990be75 (127.0.97.1:42135): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ddba503d8cdf451c94f9507f15e47737: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42135: connect: Connection refused (error 111)
I20260620 03:10:23.712167 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.704606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53762 (local address 127.0.97.2:36297)
0620 03:10:23.704796 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:23.704804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:23.704825 (+ 21us) server_negotiation.cc:407] Connection header received
0620 03:10:23.705601 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:23.705610 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:23.705719 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:23.705838 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:23.706303 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.707024 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.709146 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.709561 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.709654 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:23.710770 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:23.710778 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:23.710791 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:23.711407 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:23.711468 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:23.711487 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:23.711603 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:23.711747 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 03:10:23.711866 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:10:23.712909 18409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:23.759429 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.758831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42135 (local address 127.0.0.1:41514)
0620 03:10:23.759073 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:23.759200 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 03:10:23.760583 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.760187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42135 (local address 127.0.0.1:41530)
0620 03:10:23.760347 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:23.760427 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 03:10:23.802364 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 18287
I20260620 03:10:23.820562 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 18417
I20260620 03:10:23.821221 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.820619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42135 (local address 127.0.0.1:41532)
0620 03:10:23.820910 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:23.821013 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260620 03:10:23.822693 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.822248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42135 (local address 127.0.0.1:41538)
0620 03:10:23.822455 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:23.822536 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 03:10:23.828576 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.821968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38236 (local address 127.0.97.3:46631)
0620 03:10:23.822132 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:23.822142 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:23.822160 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:10:23.822224 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:23.822229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:23.822328 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:23.822425 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:23.823104 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.823954 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.825344 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:23.825805 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:23.825941 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:23.827046 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:23.827057 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:23.827070 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:23.827726 (+ 656us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:23.827779 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:23.827799 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:23.827927 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:23.828062 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 03:10:23.828152 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 03:10:23.829356 18539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:23.904070 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.903512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42135 (local address 127.0.0.1:41550)
0620 03:10:23.903782 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:23.903891 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 03:10:23.905193 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:23.904750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42135 (local address 127.0.0.1:41562)
0620 03:10:23.904965 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:23.905048 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260620 03:10:23.951054 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 18417
I20260620 03:10:23.973779 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 18066
I20260620 03:10:23.974825 18127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:24.083206 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:24.082543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42135 (local address 127.0.0.1:41568)
0620 03:10:24.082816 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:24.083011 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 03:10:24.083374 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 18066
2026-06-20T03:10:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2784 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:10:24 dist-test-slave-xv68 krb5kdc[18594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:10:24 dist-test-slave-xv68 krb5kdc[18594](info): set up 2 sockets
Jun 20 03:10:24 dist-test-slave-xv68 krb5kdc[18594](info): commencing operation
krb5kdc: starting...
W20260620 03:10:25.191362 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:10:25 dist-test-slave-xv68 krb5kdc[18594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:10:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:10:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:10:25.446022 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33367
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:25.768879 18607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:25.769208 18607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:25.769367 18607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:25.776410 18607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:10:25.776564 18607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:25.776669 18607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:25.776769 18607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:10:25.776852 18607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:10:25.786315 18607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:25.787624 18607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:25.788925 18607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:25.800431 18613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:25.800549 18615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:25.800746 18612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:25.801626 18607 server_base.cc:1034] running on GCE node
I20260620 03:10:25.802428 18607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:25.803843 18607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:25.805083 18607 hybrid_clock.cc:648] HybridClock initialized: now 1781925025805031 us; error 57 us; skew 500 ppm
I20260620 03:10:25.805423 18607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:25 dist-test-slave-xv68 krb5kdc[18594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:25.808598 18607 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:10:25.810096 18607 webserver.cc:460] Webserver started at http://127.0.97.62:41281/ using document root <none> and password file <none>
I20260620 03:10:25.810784 18607 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:25.810928 18607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:25.811257 18607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:25.814188 18607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "68733fa3a79e48e5b19744603a2a4368"
format_stamp: "Formatted at 2026-06-20 03:10:25 on dist-test-slave-xv68"
server_key: "7467be32987216195eb8d7a4842edc92"
server_key_iv: "d66c63300163aa7aee5206261e1e236f"
server_key_version: "encryptionkey@0"
I20260620 03:10:25.814945 18607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "68733fa3a79e48e5b19744603a2a4368"
format_stamp: "Formatted at 2026-06-20 03:10:25 on dist-test-slave-xv68"
server_key: "7467be32987216195eb8d7a4842edc92"
server_key_iv: "d66c63300163aa7aee5206261e1e236f"
server_key_version: "encryptionkey@0"
I20260620 03:10:25.820096 18607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 03:10:25.823724 18622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:25.825048 18607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 03:10:25.825241 18607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "68733fa3a79e48e5b19744603a2a4368"
format_stamp: "Formatted at 2026-06-20 03:10:25 on dist-test-slave-xv68"
server_key: "7467be32987216195eb8d7a4842edc92"
server_key_iv: "d66c63300163aa7aee5206261e1e236f"
server_key_version: "encryptionkey@0"
I20260620 03:10:25.825455 18607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:25.848489 18607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:25.853075 18607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:25.853372 18607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:25.872763 18607 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33367
I20260620 03:10:25.872817 18673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33367 every 8 connection(s)
I20260620 03:10:25.874087 18607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:10:25.877841 18674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:25.877893 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 18607
I20260620 03:10:25.878180 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:10:25.878674 388 external_mini_cluster.cc:1428] Setting key 5e4d9418b2583c337492fd8eae04f6b8
I20260620 03:10:25.891434 18674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368: Bootstrap starting.
Jun 20 03:10:25 dist-test-slave-xv68 krb5kdc[18594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:10:25.894977 18674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:25.896091 18674 log.cc:824] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:25.898844 18674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368: No bootstrap required, opened a new log
I20260620 03:10:25.902269 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:25.880565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36656 (local address 127.0.97.62:33367)
0620 03:10:25.881392 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:25.881403 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:25.881441 (+ 38us) server_negotiation.cc:407] Connection header received
0620 03:10:25.883609 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:25.883633 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:25.883988 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:25.884304 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:25.885986 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:25.887224 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:25.889166 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:25.889824 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:25.889944 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:25.894801 (+ 4857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:25.894809 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:25.894822 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:25.894860 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:25.898117 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:25.898787 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:25.898801 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:25.898807 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:25.899036 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:25.899501 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:25.899506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:25.899509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:25.900040 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:25.900393 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:25.900881 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 03:10:25.901301 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":370,"threads_started":1}
I20260620 03:10:25.906756 18674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68733fa3a79e48e5b19744603a2a4368" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33367 } }
I20260620 03:10:25.907126 18674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:25.907239 18674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68733fa3a79e48e5b19744603a2a4368, State: Initialized, Role: FOLLOWER
I20260620 03:10:25.907896 18674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68733fa3a79e48e5b19744603a2a4368" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33367 } }
I20260620 03:10:25.908134 18674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:10:25.908277 18674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:10:25.908465 18674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:25.910602 18674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68733fa3a79e48e5b19744603a2a4368" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33367 } }
I20260620 03:10:25.911079 18674 leader_election.cc:302] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68733fa3a79e48e5b19744603a2a4368; no voters:
I20260620 03:10:25.911710 18674 leader_election.cc:288] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:10:25.911839 18679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:25.912520 18679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368 [term 1 LEADER]: Becoming Leader. State: Replica: 68733fa3a79e48e5b19744603a2a4368, State: Running, Role: LEADER
I20260620 03:10:25.913057 18679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68733fa3a79e48e5b19744603a2a4368" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33367 } }
I20260620 03:10:25.913828 18674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:10:25.915810 18681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68733fa3a79e48e5b19744603a2a4368. Latest consensus state: current_term: 1 leader_uuid: "68733fa3a79e48e5b19744603a2a4368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68733fa3a79e48e5b19744603a2a4368" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33367 } } }
I20260620 03:10:25.916270 18681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:25.917796 18680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68733fa3a79e48e5b19744603a2a4368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68733fa3a79e48e5b19744603a2a4368" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33367 } } }
I20260620 03:10:25.918197 18680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:25.920759 18686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:10:25.926103 18686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:10:25.935027 18686 catalog_manager.cc:1347] Generated new cluster ID: 0d8cdfb1515742e9abc43dce7f1763ce
I20260620 03:10:25.935160 18686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:10:25.948101 18686 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:10:25.950008 18686 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:10:25.958737 18686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368: Generated new TSK 0
I20260620 03:10:25.959823 18686 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:10:26.020977 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33367
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:26.348730 18700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:26.349036 18700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:26.349212 18700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:26.356253 18700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:26.356400 18700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:26.356583 18700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:10:26.366221 18700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:26.367470 18700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:26.368757 18700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:26.381227 18705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:26.381292 18706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:26.381275 18708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:26.382050 18700 server_base.cc:1034] running on GCE node
I20260620 03:10:26.382644 18700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:26.383610 18700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:26.384838 18700 hybrid_clock.cc:648] HybridClock initialized: now 1781925026384792 us; error 35 us; skew 500 ppm
I20260620 03:10:26.385172 18700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:26 dist-test-slave-xv68 krb5kdc[18594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:26.388782 18700 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:10:26.390205 18700 webserver.cc:460] Webserver started at http://127.0.97.1:39467/ using document root <none> and password file <none>
I20260620 03:10:26.390893 18700 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:26.390996 18700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:26.391270 18700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:26.394915 18700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "5b15d2cae4bd4f4a88e781a3e9b81825"
format_stamp: "Formatted at 2026-06-20 03:10:26 on dist-test-slave-xv68"
server_key: "096f9619a091a3373c4d8b9ab48faf3a"
server_key_iv: "c1d3c116cfb68f95bbc68ceaee7eabc4"
server_key_version: "encryptionkey@0"
I20260620 03:10:26.395778 18700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5b15d2cae4bd4f4a88e781a3e9b81825"
format_stamp: "Formatted at 2026-06-20 03:10:26 on dist-test-slave-xv68"
server_key: "096f9619a091a3373c4d8b9ab48faf3a"
server_key_iv: "c1d3c116cfb68f95bbc68ceaee7eabc4"
server_key_version: "encryptionkey@0"
I20260620 03:10:26.400712 18700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 03:10:26.404647 18715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:26.405941 18700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:10:26.406127 18700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "5b15d2cae4bd4f4a88e781a3e9b81825"
format_stamp: "Formatted at 2026-06-20 03:10:26 on dist-test-slave-xv68"
server_key: "096f9619a091a3373c4d8b9ab48faf3a"
server_key_iv: "c1d3c116cfb68f95bbc68ceaee7eabc4"
server_key_version: "encryptionkey@0"
I20260620 03:10:26.406361 18700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:26.429652 18700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:26.434871 18700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:26.435200 18700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:26.436210 18700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:26.437628 18700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:26.437740 18700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:26.437908 18700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:26.437995 18700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:26.473361 18700 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45487
I20260620 03:10:26.473399 18827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45487 every 8 connection(s)
I20260620 03:10:26.474730 18700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:10:26.478179 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 18700
I20260620 03:10:26.478358 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:10:26.478824 388 external_mini_cluster.cc:1428] Setting key 2345bc338abb891d1667a1b09ea58510
Jun 20 03:10:26 dist-test-slave-xv68 krb5kdc[18594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:10:26.500741 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:26.477421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58685 (local address 127.0.97.62:33367)
0620 03:10:26.477986 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:26.477993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:26.479085 (+ 1092us) server_negotiation.cc:407] Connection header received
0620 03:10:26.481370 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:26.481377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:26.481459 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:26.481588 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:26.483123 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:26.484266 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:26.485758 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:26.486484 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:26.486631 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:26.491629 (+ 4998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:26.491634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:26.491638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:26.491687 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:26.496189 (+ 4502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:26.497363 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:26.497369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:26.497373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:26.497547 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:26.498293 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:26.498301 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:26.498308 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:26.498703 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:26.498868 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:26.499973 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0620 03:10:26.500271 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":270,"threads_started":1}
I20260620 03:10:26.502435 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:26.478021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33367 (local address 127.0.97.1:58685)
0620 03:10:26.478902 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:26.478959 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:26.481086 (+ 2127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:26.481826 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:26.481835 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:26.482372 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:10:26.482949 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:26.482966 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:26.484407 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:26.484411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:26.485615 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:26.485626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:26.486714 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:26.486719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:26.487032 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:26.488186 (+ 1154us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:10:26.488226 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:10:26.491407 (+ 3181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:10:26.496558 (+ 5151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:10:26.496570 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:10:26.496589 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:10:26.497236 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:10:26.497914 (+ 678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:10:26.497917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:10:26.497920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:10:26.498155 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:10:26.498859 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:10:26.498869 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:10:26.499490 (+ 621us) client_negotiation.cc:769] Sending connection context
0620 03:10:26.499846 (+ 356us) client_negotiation.cc:240] Negotiation successful
0620 03:10:26.500574 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":379,"threads_started":1}
I20260620 03:10:26.504426 18828 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33367
I20260620 03:10:26.504745 18828 heartbeater.cc:464] Registering TS with master...
I20260620 03:10:26.505631 18828 heartbeater.cc:511] Master 127.0.97.62:33367 requested a full tablet report, sending...
I20260620 03:10:26.508646 18639 ts_manager.cc:194] Registered new tserver with Master: 5b15d2cae4bd4f4a88e781a3e9b81825 (127.0.97.1:45487)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 03:10:26.510825 18639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:10:26.546415 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33367
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:26.865738 18834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:26.865993 18834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:26.866146 18834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:26.874493 18834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:26.874642 18834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:26.874832 18834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:10:26.884246 18834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:26.885377 18834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:26.886584 18834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:26.899544 18842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:26.899619 18840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:26.899864 18839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:26.900709 18834 server_base.cc:1034] running on GCE node
I20260620 03:10:26.901350 18834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:26.902249 18834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:26.903441 18834 hybrid_clock.cc:648] HybridClock initialized: now 1781925026903398 us; error 32 us; skew 500 ppm
I20260620 03:10:26.903756 18834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:26 dist-test-slave-xv68 krb5kdc[18594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:26.911623 18834 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:10:26.913061 18834 webserver.cc:460] Webserver started at http://127.0.97.2:34129/ using document root <none> and password file <none>
I20260620 03:10:26.913748 18834 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:26.913888 18834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:26.914230 18834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:26.917168 18834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "e5de7215beb6437dbba20e25df99f5e8"
format_stamp: "Formatted at 2026-06-20 03:10:26 on dist-test-slave-xv68"
server_key: "c3bde1b0a0483a0343fc681f363e3efb"
server_key_iv: "40f5d84c62ab4c028fde242decb46d9f"
server_key_version: "encryptionkey@0"
I20260620 03:10:26.917943 18834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e5de7215beb6437dbba20e25df99f5e8"
format_stamp: "Formatted at 2026-06-20 03:10:26 on dist-test-slave-xv68"
server_key: "c3bde1b0a0483a0343fc681f363e3efb"
server_key_iv: "40f5d84c62ab4c028fde242decb46d9f"
server_key_version: "encryptionkey@0"
I20260620 03:10:26.922928 18834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 03:10:26.926409 18849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:26.927851 18834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:10:26.928040 18834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "e5de7215beb6437dbba20e25df99f5e8"
format_stamp: "Formatted at 2026-06-20 03:10:26 on dist-test-slave-xv68"
server_key: "c3bde1b0a0483a0343fc681f363e3efb"
server_key_iv: "40f5d84c62ab4c028fde242decb46d9f"
server_key_version: "encryptionkey@0"
I20260620 03:10:26.928248 18834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:26.945683 18834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:26.949755 18834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:26.950038 18834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:26.950970 18834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:26.952306 18834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:26.952409 18834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:26.952562 18834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:26.952639 18834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:26.985713 18834 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34115
I20260620 03:10:26.985796 18961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34115 every 8 connection(s)
I20260620 03:10:26.986949 18834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:10:26.988674 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 18834
I20260620 03:10:26.988888 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:10:26.989390 388 external_mini_cluster.cc:1428] Setting key e997cb9a8a62102969d642351c1414d1
Jun 20 03:10:26 dist-test-slave-xv68 krb5kdc[18594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:10:27.010538 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:26.989673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43653 (local address 127.0.97.62:33367)
0620 03:10:26.989899 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:26.989905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:26.991838 (+ 1933us) server_negotiation.cc:407] Connection header received
0620 03:10:26.993773 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:26.993779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:26.993869 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:26.994000 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:26.995681 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:26.996475 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:26.999028 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:26.999884 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.000045 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:27.004171 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:27.004177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:27.004180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:27.004214 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:27.006758 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:27.007845 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:27.007850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:27.007857 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:27.008000 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:27.008591 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:27.008596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:27.008599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:27.008889 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:27.009056 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:27.010037 (+ 981us) server_negotiation.cc:299] Negotiation successful
0620 03:10:27.010237 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 03:10:27.012207 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:26.990149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33367 (local address 127.0.97.2:43653)
0620 03:10:26.991664 (+ 1515us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:26.991710 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:26.993481 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:26.994245 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:26.994254 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:26.994818 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:10:26.995479 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:26.995499 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:26.996691 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:26.996700 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:26.998860 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:26.998874 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.000105 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.000111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:27.000301 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:27.001003 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:10:27.001028 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:10:27.003940 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:10:27.006903 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:10:27.006916 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:10:27.006938 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:10:27.007686 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:10:27.008127 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:10:27.008136 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:10:27.008144 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:10:27.008421 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:10:27.009044 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:10:27.009057 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:10:27.009584 (+ 527us) client_negotiation.cc:769] Sending connection context
0620 03:10:27.009911 (+ 327us) client_negotiation.cc:240] Negotiation successful
0620 03:10:27.010543 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1205,"thread_start_us":934,"threads_started":1}
I20260620 03:10:27.014027 18962 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33367
I20260620 03:10:27.014421 18962 heartbeater.cc:464] Registering TS with master...
I20260620 03:10:27.015228 18962 heartbeater.cc:511] Master 127.0.97.62:33367 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 03:10:27.016819 18639 ts_manager.cc:194] Registered new tserver with Master: e5de7215beb6437dbba20e25df99f5e8 (127.0.97.2:34115)
I20260620 03:10:27.017755 18639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:10:27.048032 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33367
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:27.363588 18967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:27.363920 18967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:27.364089 18967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:27.371254 18967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:27.371415 18967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:27.371585 18967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:10:27.380717 18967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:27.381907 18967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:27.383139 18967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:27.396731 18972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:27.397264 18975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:27.397632 18973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:27.401731 18967 server_base.cc:1034] running on GCE node
I20260620 03:10:27.402540 18967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:27.403875 18967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:27.405103 18967 hybrid_clock.cc:648] HybridClock initialized: now 1781925027404982 us; error 91 us; skew 500 ppm
I20260620 03:10:27.405465 18967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:27 dist-test-slave-xv68 krb5kdc[18594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:27.409572 18967 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:10:27.411608 18967 webserver.cc:460] Webserver started at http://127.0.97.3:35635/ using document root <none> and password file <none>
I20260620 03:10:27.412529 18967 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:27.412668 18967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:27.413033 18967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:27.417081 18967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "d2ab081a5b5d43478c569696b7734bc9"
format_stamp: "Formatted at 2026-06-20 03:10:27 on dist-test-slave-xv68"
server_key: "abb75ebc675aaba0a181d2a6e9ad828f"
server_key_iv: "00ef3e8d00d368b51714591d8a62e9ac"
server_key_version: "encryptionkey@0"
I20260620 03:10:27.417907 18967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d2ab081a5b5d43478c569696b7734bc9"
format_stamp: "Formatted at 2026-06-20 03:10:27 on dist-test-slave-xv68"
server_key: "abb75ebc675aaba0a181d2a6e9ad828f"
server_key_iv: "00ef3e8d00d368b51714591d8a62e9ac"
server_key_version: "encryptionkey@0"
I20260620 03:10:27.424229 18967 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260620 03:10:27.427611 18982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:27.428833 18967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:10:27.428998 18967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "d2ab081a5b5d43478c569696b7734bc9"
format_stamp: "Formatted at 2026-06-20 03:10:27 on dist-test-slave-xv68"
server_key: "abb75ebc675aaba0a181d2a6e9ad828f"
server_key_iv: "00ef3e8d00d368b51714591d8a62e9ac"
server_key_version: "encryptionkey@0"
I20260620 03:10:27.429211 18967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:27.448909 18967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:27.452817 18967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:27.453084 18967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:27.454010 18967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:27.455269 18967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:27.455397 18967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:27.455560 18967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:27.455610 18967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:27.490142 18967 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38175
I20260620 03:10:27.490296 19094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38175 every 8 connection(s)
I20260620 03:10:27.491426 18967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:10:27.500547 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 18967
I20260620 03:10:27.500749 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:10:27.501227 388 external_mini_cluster.cc:1428] Setting key 819d74964d70818a8babf88cc387a8a5
Jun 20 03:10:27 dist-test-slave-xv68 krb5kdc[18594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:10:27.510222 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:27.493867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39357 (local address 127.0.97.62:33367)
0620 03:10:27.494110 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:27.494117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:27.495174 (+ 1057us) server_negotiation.cc:407] Connection header received
0620 03:10:27.496996 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:27.497000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:27.497065 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:27.497159 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:27.498527 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.499370 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.500906 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.501338 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.501459 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:27.504903 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:27.504907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:27.504911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:27.504942 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:27.507363 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:27.508021 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:27.508026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:27.508029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:27.508176 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:27.508537 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:27.508540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:27.508548 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:27.508859 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:27.508980 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:27.509576 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 03:10:27.509897 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 03:10:27.510887 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:27.494330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33367 (local address 127.0.97.3:39357)
0620 03:10:27.495020 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:27.495058 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:27.496756 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:27.497344 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:27.497351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:27.497872 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:10:27.498381 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:27.498397 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.499490 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.499493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:27.500752 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:27.500764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.501541 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.501550 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:27.501783 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:27.502429 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:10:27.502455 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:10:27.504715 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:10:27.507484 (+ 2769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:10:27.507490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:10:27.507500 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:10:27.507888 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:10:27.508283 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:10:27.508286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:10:27.508288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:10:27.508443 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:10:27.508969 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:10:27.508974 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:10:27.509257 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 03:10:27.509467 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 03:10:27.509810 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":287,"threads_started":1}
I20260620 03:10:27.512280 19095 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33367
I20260620 03:10:27.512589 19095 heartbeater.cc:464] Registering TS with master...
I20260620 03:10:27.513316 19095 heartbeater.cc:511] Master 127.0.97.62:33367 requested a full tablet report, sending...
I20260620 03:10:27.514451 18828 heartbeater.cc:503] Master 127.0.97.62:33367 was elected leader, sending a full tablet report...
I20260620 03:10:27.514962 18639 ts_manager.cc:194] Registered new tserver with Master: d2ab081a5b5d43478c569696b7734bc9 (127.0.97.3:38175)
I20260620 03:10:27.515903 18639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39357
I20260620 03:10:27.516347 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:10:27.534149 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:27.521114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49726 (local address 127.0.97.62:33367)
0620 03:10:27.521271 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:27.521276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:27.521698 (+ 422us) server_negotiation.cc:407] Connection header received
0620 03:10:27.521845 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:27.521849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:27.521902 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:27.521982 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:27.523359 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.524073 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.525357 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.525779 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.525869 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:27.528484 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:27.528489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:27.528493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:27.528524 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:27.531399 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:27.532203 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:27.532214 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:27.532217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:27.532329 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:27.532743 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:27.532748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:27.532750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:27.533142 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:27.533372 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:27.533708 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 03:10:27.533906 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 03:10:27.537832 18639 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:10:27.544507 18639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:10:27.578684 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:27.555599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38175 (local address 127.0.0.1:60302)
0620 03:10:27.568822 (+ 13223us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:27.568893 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:27.569168 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:27.571365 (+ 2197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:27.571374 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:27.571403 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:27.571924 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:27.571939 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.574377 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.574382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:27.576625 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:27.576636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.577975 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.577980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:27.578131 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:27.578179 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 03:10:27.578279 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 03:10:27.578398 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12882,"mutex_wait_us":1,"thread_start_us":14626,"threads_started":1}
I20260620 03:10:27.578861 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:27.555881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.0.97.3:38175)
0620 03:10:27.570681 (+ 14800us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:27.570693 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:27.570732 (+ 39us) server_negotiation.cc:407] Connection header received
0620 03:10:27.570838 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:27.570848 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:27.571102 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:27.572167 (+ 1065us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:27.572463 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.574194 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.576776 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.577789 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.577910 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:27.578152 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:27.578365 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 03:10:27.578530 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14620,"thread_start_us":14475,"threads_started":1}
I20260620 03:10:27.580229 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:27.570934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43076 (local address 127.0.97.2:34115)
0620 03:10:27.571523 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:27.571531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:27.572331 (+ 800us) server_negotiation.cc:407] Connection header received
0620 03:10:27.572620 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:27.572627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:27.572822 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:27.572982 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:27.573651 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.575018 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.577665 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.579084 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.579217 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:27.579435 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:27.579742 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 03:10:27.579905 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":334,"threads_started":1}
I20260620 03:10:27.580943 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:27.571580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34115 (local address 127.0.0.1:43076)
0620 03:10:27.572214 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:27.572238 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:27.572430 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:27.573222 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:27.573225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:27.573243 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:27.573493 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:27.573499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.575140 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.575145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:27.577475 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:27.577488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.579286 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.579342 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:27.579524 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:27.579561 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 03:10:27.580420 (+ 859us) client_negotiation.cc:240] Negotiation successful
0620 03:10:27.580548 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":26,"thread_start_us":322,"threads_started":1}
I20260620 03:10:27.582202 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:27.555916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45487 (local address 127.0.0.1:48730)
0620 03:10:27.569947 (+ 14031us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:27.569978 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:27.570124 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:27.571873 (+ 1749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:27.571880 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:27.571909 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:27.572399 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:27.572415 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.574297 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.574305 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:27.577473 (+ 3168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:27.577488 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.581423 (+ 3935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.581432 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:27.581621 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:27.581652 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 03:10:27.581745 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 03:10:27.581859 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13863,"mutex_wait_us":75,"thread_start_us":13491,"threads_started":1}
I20260620 03:10:27.582319 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:27.556642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48730 (local address 127.0.97.1:45487)
0620 03:10:27.571037 (+ 14395us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:27.571044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:27.571072 (+ 28us) server_negotiation.cc:407] Connection header received
0620 03:10:27.571163 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:27.571169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:27.571409 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:27.571596 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:27.572602 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.574141 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.579405 (+ 5264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.580967 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.581096 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:27.581301 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:27.581819 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 03:10:27.581978 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14242,"thread_start_us":14080,"threads_started":1}
I20260620 03:10:27.583256 19030 tablet_service.cc:1467] Processing CreateTablet for tablet 82588995d4b44a72b9c70be9681ca467 (DEFAULT_TABLE table=test-table [id=20766fd482ef43369b142453ce5731df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:10:27.584336 18897 tablet_service.cc:1467] Processing CreateTablet for tablet 82588995d4b44a72b9c70be9681ca467 (DEFAULT_TABLE table=test-table [id=20766fd482ef43369b142453ce5731df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:10:27.584833 19030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82588995d4b44a72b9c70be9681ca467. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:27.585510 18897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82588995d4b44a72b9c70be9681ca467. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:27.585811 18763 tablet_service.cc:1467] Processing CreateTablet for tablet 82588995d4b44a72b9c70be9681ca467 (DEFAULT_TABLE table=test-table [id=20766fd482ef43369b142453ce5731df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:10:27.587064 18763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82588995d4b44a72b9c70be9681ca467. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:27.596881 19113 tablet_bootstrap.cc:492] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8: Bootstrap starting.
I20260620 03:10:27.599223 19114 tablet_bootstrap.cc:492] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9: Bootstrap starting.
I20260620 03:10:27.600528 19113 tablet_bootstrap.cc:654] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:27.601389 19115 tablet_bootstrap.cc:492] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825: Bootstrap starting.
I20260620 03:10:27.602203 19113 log.cc:824] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:27.603181 19114 tablet_bootstrap.cc:654] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:27.604465 19115 tablet_bootstrap.cc:654] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:27.604717 19114 log.cc:824] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:27.605511 19115 log.cc:824] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:27.608306 19114 tablet_bootstrap.cc:492] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9: No bootstrap required, opened a new log
I20260620 03:10:27.608583 19114 ts_tablet_manager.cc:1397] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 03:10:27.609840 19113 tablet_bootstrap.cc:492] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8: No bootstrap required, opened a new log
I20260620 03:10:27.609844 19115 tablet_bootstrap.cc:492] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825: No bootstrap required, opened a new log
I20260620 03:10:27.610136 19113 ts_tablet_manager.cc:1397] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260620 03:10:27.610155 19115 ts_tablet_manager.cc:1397] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260620 03:10:27.616525 19114 raft_consensus.cc:348] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab081a5b5d43478c569696b7734bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38175 } } peers { permanent_uuid: "5b15d2cae4bd4f4a88e781a3e9b81825" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45487 } } peers { permanent_uuid: "e5de7215beb6437dbba20e25df99f5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34115 } }
I20260620 03:10:27.616892 19114 raft_consensus.cc:374] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:27.616997 19114 raft_consensus.cc:729] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ab081a5b5d43478c569696b7734bc9, State: Initialized, Role: FOLLOWER
I20260620 03:10:27.617640 19114 consensus_queue.cc:260] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab081a5b5d43478c569696b7734bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38175 } } peers { permanent_uuid: "5b15d2cae4bd4f4a88e781a3e9b81825" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45487 } } peers { permanent_uuid: "e5de7215beb6437dbba20e25df99f5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34115 } }
I20260620 03:10:27.618739 19095 heartbeater.cc:503] Master 127.0.97.62:33367 was elected leader, sending a full tablet report...
I20260620 03:10:27.619415 19114 ts_tablet_manager.cc:1428] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260620 03:10:27.620926 19113 raft_consensus.cc:348] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab081a5b5d43478c569696b7734bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38175 } } peers { permanent_uuid: "5b15d2cae4bd4f4a88e781a3e9b81825" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45487 } } peers { permanent_uuid: "e5de7215beb6437dbba20e25df99f5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34115 } }
I20260620 03:10:27.620927 19115 raft_consensus.cc:348] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab081a5b5d43478c569696b7734bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38175 } } peers { permanent_uuid: "5b15d2cae4bd4f4a88e781a3e9b81825" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45487 } } peers { permanent_uuid: "e5de7215beb6437dbba20e25df99f5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34115 } }
I20260620 03:10:27.621393 19113 raft_consensus.cc:374] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:27.621429 19115 raft_consensus.cc:374] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:27.621518 19113 raft_consensus.cc:729] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5de7215beb6437dbba20e25df99f5e8, State: Initialized, Role: FOLLOWER
I20260620 03:10:27.621515 19115 raft_consensus.cc:729] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b15d2cae4bd4f4a88e781a3e9b81825, State: Initialized, Role: FOLLOWER
I20260620 03:10:27.622097 19115 consensus_queue.cc:260] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab081a5b5d43478c569696b7734bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38175 } } peers { permanent_uuid: "5b15d2cae4bd4f4a88e781a3e9b81825" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45487 } } peers { permanent_uuid: "e5de7215beb6437dbba20e25df99f5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34115 } }
I20260620 03:10:27.622107 19113 consensus_queue.cc:260] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab081a5b5d43478c569696b7734bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38175 } } peers { permanent_uuid: "5b15d2cae4bd4f4a88e781a3e9b81825" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45487 } } peers { permanent_uuid: "e5de7215beb6437dbba20e25df99f5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34115 } }
I20260620 03:10:27.624015 18962 heartbeater.cc:503] Master 127.0.97.62:33367 was elected leader, sending a full tablet report...
I20260620 03:10:27.626411 19113 ts_tablet_manager.cc:1428] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
I20260620 03:10:27.628383 19115 ts_tablet_manager.cc:1428] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825: Time spent starting tablet: real 0.018s user 0.008s sys 0.009s
I20260620 03:10:27.662716 19120 raft_consensus.cc:482] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:10:27.662981 19120 raft_consensus.cc:504] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab081a5b5d43478c569696b7734bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38175 } } peers { permanent_uuid: "5b15d2cae4bd4f4a88e781a3e9b81825" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45487 } } peers { permanent_uuid: "e5de7215beb6437dbba20e25df99f5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34115 } }
I20260620 03:10:27.664645 19120 leader_election.cc:288] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2ab081a5b5d43478c569696b7734bc9 (127.0.97.3:38175), 5b15d2cae4bd4f4a88e781a3e9b81825 (127.0.97.1:45487)
I20260620 03:10:27.673208 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:27.665053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41335 (local address 127.0.97.3:38175)
0620 03:10:27.665350 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:27.665358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:27.666139 (+ 781us) server_negotiation.cc:407] Connection header received
0620 03:10:27.666367 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:27.666373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:27.666453 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:27.666557 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:27.667358 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.668520 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.671573 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.672329 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.672430 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:27.672504 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:27.672824 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 03:10:27.672957 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:10:27.673318 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:27.664957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38175 (local address 127.0.97.2:41335)
0620 03:10:27.665957 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:27.665977 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:27.666189 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:27.666596 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:27.666601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:27.666630 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:27.667177 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:27.667186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.668697 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.668702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:27.671407 (+ 2705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:27.671424 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.672515 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.672519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:27.672651 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:27.672676 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 03:10:27.672785 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 03:10:27.672919 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":246,"spinlock_wait_cycles":34048,"thread_start_us":636,"threads_started":1}
I20260620 03:10:27.674422 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:27.665142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50035 (local address 127.0.97.1:45487)
0620 03:10:27.665335 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:27.665341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:27.666135 (+ 794us) server_negotiation.cc:407] Connection header received
0620 03:10:27.666367 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:27.666373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:27.666453 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:27.666557 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:27.667358 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.668520 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.672576 (+ 4056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.673442 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.673564 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:27.673663 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:27.674155 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 03:10:27.674259 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:10:27.674263 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82588995d4b44a72b9c70be9681ca467" candidate_uuid: "e5de7215beb6437dbba20e25df99f5e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2ab081a5b5d43478c569696b7734bc9" is_pre_election: true
I20260620 03:10:27.674646 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:27.664996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45487 (local address 127.0.97.2:50035)
0620 03:10:27.666011 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:27.666027 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:27.666147 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:27.666619 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:27.666623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:27.666638 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:27.667125 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:27.667140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.668697 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.668702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:27.671407 (+ 2705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:27.671424 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.673764 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.673770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:27.673953 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:27.673990 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 03:10:27.674107 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 03:10:27.674231 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":129,"thread_start_us":554,"threads_started":1}
I20260620 03:10:27.674830 19050 raft_consensus.cc:2454] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5de7215beb6437dbba20e25df99f5e8 in term 0.
I20260620 03:10:27.675590 18852 leader_election.cc:302] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2ab081a5b5d43478c569696b7734bc9, e5de7215beb6437dbba20e25df99f5e8; no voters:
I20260620 03:10:27.675595 18783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82588995d4b44a72b9c70be9681ca467" candidate_uuid: "e5de7215beb6437dbba20e25df99f5e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b15d2cae4bd4f4a88e781a3e9b81825" is_pre_election: true
I20260620 03:10:27.675951 19120 raft_consensus.cc:2781] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:10:27.676100 19120 raft_consensus.cc:482] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:10:27.676218 19120 raft_consensus.cc:3037] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:27.676180 18783 raft_consensus.cc:2454] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5de7215beb6437dbba20e25df99f5e8 in term 0.
I20260620 03:10:27.678491 19120 raft_consensus.cc:504] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab081a5b5d43478c569696b7734bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38175 } } peers { permanent_uuid: "5b15d2cae4bd4f4a88e781a3e9b81825" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45487 } } peers { permanent_uuid: "e5de7215beb6437dbba20e25df99f5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34115 } }
I20260620 03:10:27.679140 19120 leader_election.cc:288] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [CANDIDATE]: Term 1 election: Requested vote from peers d2ab081a5b5d43478c569696b7734bc9 (127.0.97.3:38175), 5b15d2cae4bd4f4a88e781a3e9b81825 (127.0.97.1:45487)
I20260620 03:10:27.679558 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82588995d4b44a72b9c70be9681ca467" candidate_uuid: "e5de7215beb6437dbba20e25df99f5e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2ab081a5b5d43478c569696b7734bc9"
I20260620 03:10:27.679682 18783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82588995d4b44a72b9c70be9681ca467" candidate_uuid: "e5de7215beb6437dbba20e25df99f5e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b15d2cae4bd4f4a88e781a3e9b81825"
I20260620 03:10:27.679780 19050 raft_consensus.cc:3037] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:27.679873 18783 raft_consensus.cc:3037] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:27.682966 19050 raft_consensus.cc:2454] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5de7215beb6437dbba20e25df99f5e8 in term 1.
I20260620 03:10:27.682966 18783 raft_consensus.cc:2454] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5de7215beb6437dbba20e25df99f5e8 in term 1.
I20260620 03:10:27.683388 18850 leader_election.cc:302] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b15d2cae4bd4f4a88e781a3e9b81825, e5de7215beb6437dbba20e25df99f5e8; no voters:
I20260620 03:10:27.683665 19120 raft_consensus.cc:2781] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:27.684091 19120 raft_consensus.cc:686] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [term 1 LEADER]: Becoming Leader. State: Replica: e5de7215beb6437dbba20e25df99f5e8, State: Running, Role: LEADER
I20260620 03:10:27.684629 19120 consensus_queue.cc:237] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab081a5b5d43478c569696b7734bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38175 } } peers { permanent_uuid: "5b15d2cae4bd4f4a88e781a3e9b81825" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45487 } } peers { permanent_uuid: "e5de7215beb6437dbba20e25df99f5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34115 } }
I20260620 03:10:27.689519 18638 catalog_manager.cc:5495] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5de7215beb6437dbba20e25df99f5e8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e5de7215beb6437dbba20e25df99f5e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab081a5b5d43478c569696b7734bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b15d2cae4bd4f4a88e781a3e9b81825" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5de7215beb6437dbba20e25df99f5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34115 } health_report { overall_health: HEALTHY } } }
I20260620 03:10:27.706156 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:27.701222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43082 (local address 127.0.97.2:34115)
0620 03:10:27.701382 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:27.701386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:27.701402 (+ 16us) server_negotiation.cc:407] Connection header received
0620 03:10:27.701549 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:27.701553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:27.701654 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:27.701769 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:10:27.702103 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.702831 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.704279 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.704689 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.704785 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:27.705218 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:10:27.705680 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:10:27.705801 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:27.705924 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 03:10:27.705959 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 03:10:27.714039 19050 raft_consensus.cc:1261] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9 [term 1 FOLLOWER]: Refusing update from remote peer e5de7215beb6437dbba20e25df99f5e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:10:27.714020 18783 raft_consensus.cc:1261] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825 [term 1 FOLLOWER]: Refusing update from remote peer e5de7215beb6437dbba20e25df99f5e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:10:27.715075 19120 consensus_queue.cc:1035] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b15d2cae4bd4f4a88e781a3e9b81825" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:10:27.715474 19124 consensus_queue.cc:1035] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2ab081a5b5d43478c569696b7734bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:10:27.726832 19127 mvcc.cc:205] Tried to move back new op lower bound from 7298764913506217984 to 7298764913400082432. Current Snapshot: MvccSnapshot[applied={T|T < 7298764913506217984}]
W20260620 03:10:27.727556 18829 tablet.cc:2369] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:10:27.731683 19128 mvcc.cc:205] Tried to move back new op lower bound from 7298764913506217984 to 7298764913400082432. Current Snapshot: MvccSnapshot[applied={T|T < 7298764913506217984}]
I20260620 03:10:27.744410 18639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:10:27.744704 18639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:10:27.747994 18639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68733fa3a79e48e5b19744603a2a4368: Sending DeleteTablet for 3 replicas of tablet 82588995d4b44a72b9c70be9681ca467
I20260620 03:10:27.749111 18763 tablet_service.cc:1514] Processing DeleteTablet for tablet 82588995d4b44a72b9c70be9681ca467 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:10:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48730
I20260620 03:10:27.749122 19030 tablet_service.cc:1514] Processing DeleteTablet for tablet 82588995d4b44a72b9c70be9681ca467 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:10:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60302
I20260620 03:10:27.751391 18897 tablet_service.cc:1514] Processing DeleteTablet for tablet 82588995d4b44a72b9c70be9681ca467 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:10:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43076
I20260620 03:10:27.753894 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 18700
I20260620 03:10:27.761507 19138 tablet_replica.cc:330] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825: stopping tablet replica
I20260620 03:10:27.762017 19138 raft_consensus.cc:2229] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:10:27.762555 19138 raft_consensus.cc:2258] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:10:27.764302 19140 tablet_replica.cc:330] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8: stopping tablet replica
I20260620 03:10:27.764683 19140 raft_consensus.cc:2229] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:10:27.765254 19138 ts_tablet_manager.cc:1905] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:10:27.765292 19140 raft_consensus.cc:2258] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:10:27.766021 19139 tablet_replica.cc:330] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9: stopping tablet replica
I20260620 03:10:27.766420 19139 raft_consensus.cc:2229] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:10:27.766894 19139 raft_consensus.cc:2258] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:10:27.767650 19140 ts_tablet_manager.cc:1905] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:10:27.772560 19139 ts_tablet_manager.cc:1905] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 03:10:27 dist-test-slave-xv68 krb5kdc[18594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:10:27.774840 19140 ts_tablet_manager.cc:1918] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:10:27.775079 19140 log.cc:1196] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/82588995d4b44a72b9c70be9681ca467
I20260620 03:10:27.775677 19140 ts_tablet_manager.cc:1939] T 82588995d4b44a72b9c70be9681ca467 P e5de7215beb6437dbba20e25df99f5e8: Deleting consensus metadata
I20260620 03:10:27.777554 18625 catalog_manager.cc:4841] TS e5de7215beb6437dbba20e25df99f5e8 (127.0.97.2:34115): tablet 82588995d4b44a72b9c70be9681ca467 (table test-table [id=20766fd482ef43369b142453ce5731df]) successfully deleted
I20260620 03:10:27.780778 19139 ts_tablet_manager.cc:1918] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:10:27.780781 19138 ts_tablet_manager.cc:1918] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:10:27.780930 19138 log.cc:1196] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/82588995d4b44a72b9c70be9681ca467
I20260620 03:10:27.781010 19139 log.cc:1196] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/82588995d4b44a72b9c70be9681ca467
I20260620 03:10:27.781540 19138 ts_tablet_manager.cc:1939] T 82588995d4b44a72b9c70be9681ca467 P 5b15d2cae4bd4f4a88e781a3e9b81825: Deleting consensus metadata
I20260620 03:10:27.781574 19139 ts_tablet_manager.cc:1939] T 82588995d4b44a72b9c70be9681ca467 P d2ab081a5b5d43478c569696b7734bc9: Deleting consensus metadata
I20260620 03:10:27.783239 18625 catalog_manager.cc:4841] TS d2ab081a5b5d43478c569696b7734bc9 (127.0.97.3:38175): tablet 82588995d4b44a72b9c70be9681ca467 (table test-table [id=20766fd482ef43369b142453ce5731df]) successfully deleted
I20260620 03:10:27.783244 18623 catalog_manager.cc:4841] TS 5b15d2cae4bd4f4a88e781a3e9b81825 (127.0.97.1:45487): tablet 82588995d4b44a72b9c70be9681ca467 (table test-table [id=20766fd482ef43369b142453ce5731df]) successfully deleted
I20260620 03:10:27.783704 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:27.755619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48746 (local address 127.0.97.1:45487)
0620 03:10:27.755833 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:27.755842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:27.763716 (+ 7874us) server_negotiation.cc:407] Connection header received
0620 03:10:27.763911 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:27.763919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:27.764014 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:27.764114 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:27.766173 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.767258 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.769168 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.769808 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.769896 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:27.775264 (+ 5368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:27.775279 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:27.775312 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:27.775354 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:27.779841 (+ 4487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:27.780652 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:27.780660 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:27.780664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:27.780827 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:27.781348 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:27.781353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:27.781355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:27.781832 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:27.782117 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:27.782715 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 03:10:27.782919 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 03:10:27.784482 18823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:27.875185 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 18700
I20260620 03:10:27.893779 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 18834
Jun 20 03:10:27 dist-test-slave-xv68 krb5kdc[18594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:10:27.909862 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:27.894844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.0.97.2:34115)
0620 03:10:27.895049 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:27.895054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:27.895074 (+ 20us) server_negotiation.cc:407] Connection header received
0620 03:10:27.895147 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:27.895153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:27.895246 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:27.895384 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:27.896715 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.897497 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.898894 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:27.899392 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:27.899538 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:27.903927 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:27.903940 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:27.903955 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:27.903991 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:27.906964 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:27.907737 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:27.907744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:27.907746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:27.907894 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:27.908299 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:27.908305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:27.908310 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:27.908689 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:27.909010 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:27.909297 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 03:10:27.909515 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 03:10:27.910533 18957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:28.024732 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 18834
I20260620 03:10:28.043236 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 18967
Jun 20 03:10:28 dist-test-slave-xv68 krb5kdc[18594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:10:28.058607 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:28.044099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60310 (local address 127.0.97.3:38175)
0620 03:10:28.044344 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:28.044350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:28.044373 (+ 23us) server_negotiation.cc:407] Connection header received
0620 03:10:28.044498 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:28.044503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:28.044591 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:28.044678 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:28.045939 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:28.046685 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:28.048100 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:28.048489 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:28.048581 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:28.052977 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:28.052992 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:28.053009 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:28.053041 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:28.055850 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:28.056461 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:28.056469 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:28.056471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:28.056602 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:28.057164 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:28.057168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:28.057171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:28.057536 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:28.057764 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:28.058051 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 03:10:28.058233 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 03:10:28.059267 19090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:28.147756 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 18967
I20260620 03:10:28.164108 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 18607
I20260620 03:10:28.165184 18669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:28.277014 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 18607
2026-06-20T03:10:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4198 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T03:10:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:10:28Z Disabled control of system clock
I20260620 03:10:28.352703 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44551
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 03:10:28.680852 19149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:28.681133 19149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:28.681265 19149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:28.688683 19149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:10:28.688799 19149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:28.688869 19149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:28.688970 19149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:10:28.689033 19149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:10:28.699133 19149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44551
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:28.700484 19149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:28.702253 19149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:28.714044 19155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:28.714164 19154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:28.714131 19157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:28.714888 19149 server_base.cc:1034] running on GCE node
I20260620 03:10:28.715790 19149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:28.717206 19149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:28.718457 19149 hybrid_clock.cc:648] HybridClock initialized: now 1781925028718412 us; error 57 us; skew 500 ppm
I20260620 03:10:28.718847 19149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:10:28.721349 19149 webserver.cc:460] Webserver started at http://127.0.97.62:33837/ using document root <none> and password file <none>
I20260620 03:10:28.722033 19149 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:28.722169 19149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:28.722492 19149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:28.725378 19149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "bb8d749f069745209efcb8407d8df240"
format_stamp: "Formatted at 2026-06-20 03:10:28 on dist-test-slave-xv68"
server_key: "2a891b00c4a22c8a4f4067a4264b5cfa"
server_key_iv: "39f353289f3f06acc084446251f8cbe4"
server_key_version: "encryptionkey@0"
I20260620 03:10:28.726173 19149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "bb8d749f069745209efcb8407d8df240"
format_stamp: "Formatted at 2026-06-20 03:10:28 on dist-test-slave-xv68"
server_key: "2a891b00c4a22c8a4f4067a4264b5cfa"
server_key_iv: "39f353289f3f06acc084446251f8cbe4"
server_key_version: "encryptionkey@0"
I20260620 03:10:28.731194 19149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 03:10:28.734622 19163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:28.735965 19149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:10:28.736153 19149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "bb8d749f069745209efcb8407d8df240"
format_stamp: "Formatted at 2026-06-20 03:10:28 on dist-test-slave-xv68"
server_key: "2a891b00c4a22c8a4f4067a4264b5cfa"
server_key_iv: "39f353289f3f06acc084446251f8cbe4"
server_key_version: "encryptionkey@0"
I20260620 03:10:28.736371 19149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:28.750933 19149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:28.751725 19149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:28.752018 19149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:28.770839 19149 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44551
I20260620 03:10:28.770859 19214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44551 every 8 connection(s)
I20260620 03:10:28.772297 19149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:10:28.776094 19215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:28.778241 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 19149
I20260620 03:10:28.778420 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:10:28.778846 388 external_mini_cluster.cc:1428] Setting key 00a3312aee8806a0656a4d8e0c6176d0
I20260620 03:10:28.790513 19215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240: Bootstrap starting.
I20260620 03:10:28.794416 19215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:28.795578 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:28.780909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51324 (local address 127.0.97.62:44551)
0620 03:10:28.782034 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:28.782051 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:28.782110 (+ 59us) server_negotiation.cc:407] Connection header received
0620 03:10:28.784514 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:28.784541 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:28.785116 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:28.785563 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:28.786138 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:28.788764 (+ 2626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:28.790204 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:28.791266 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:28.791474 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:28.792705 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:28.792716 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:28.792734 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:28.793410 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:28.793538 (+ 128us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:28.793578 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:28.793688 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:28.793903 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 03:10:28.794232 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":335,"threads_started":1}
I20260620 03:10:28.795761 19215 log.cc:824] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:28.799118 19215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240: No bootstrap required, opened a new log
I20260620 03:10:28.806542 19215 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8d749f069745209efcb8407d8df240" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44551 } }
I20260620 03:10:28.806882 19215 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:28.806952 19215 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8d749f069745209efcb8407d8df240, State: Initialized, Role: FOLLOWER
I20260620 03:10:28.807493 19215 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8d749f069745209efcb8407d8df240" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44551 } }
I20260620 03:10:28.807705 19215 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:10:28.807794 19215 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:10:28.807915 19215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:28.810314 19215 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8d749f069745209efcb8407d8df240" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44551 } }
I20260620 03:10:28.810863 19215 leader_election.cc:302] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb8d749f069745209efcb8407d8df240; no voters:
I20260620 03:10:28.811441 19215 leader_election.cc:288] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:10:28.811563 19220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:28.812296 19220 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240 [term 1 LEADER]: Becoming Leader. State: Replica: bb8d749f069745209efcb8407d8df240, State: Running, Role: LEADER
I20260620 03:10:28.812829 19220 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8d749f069745209efcb8407d8df240" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44551 } }
I20260620 03:10:28.813480 19215 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:10:28.815564 19221 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb8d749f069745209efcb8407d8df240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8d749f069745209efcb8407d8df240" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44551 } } }
I20260620 03:10:28.815570 19222 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb8d749f069745209efcb8407d8df240. Latest consensus state: current_term: 1 leader_uuid: "bb8d749f069745209efcb8407d8df240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8d749f069745209efcb8407d8df240" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44551 } } }
I20260620 03:10:28.816480 19222 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:28.817031 19221 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:28.826767 19227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:10:28.832471 19227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:10:28.841519 19227 catalog_manager.cc:1347] Generated new cluster ID: 0b54838b815a40519c4d89284e7e32c1
I20260620 03:10:28.841641 19227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:10:28.852582 19227 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:10:28.853808 19227 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:10:28.860070 19227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240: Generated new TSK 0
I20260620 03:10:28.861071 19227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:10:28.867944 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44551
--builtin_ntp_servers=127.0.97.20:37065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 03:10:29.191911 19239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:29.192178 19239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:29.192267 19239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:29.199427 19239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:29.199525 19239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:29.199632 19239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:10:29.208688 19239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:29.209929 19239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:29.211642 19239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:29.224797 19245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:29.224875 19244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:29.224875 19247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:29.225782 19239 server_base.cc:1034] running on GCE node
I20260620 03:10:29.226486 19239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:29.227641 19239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:29.228910 19239 hybrid_clock.cc:648] HybridClock initialized: now 1781925029228844 us; error 52 us; skew 500 ppm
I20260620 03:10:29.229254 19239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:10:29.231748 19239 webserver.cc:460] Webserver started at http://127.0.97.1:45211/ using document root <none> and password file <none>
I20260620 03:10:29.232430 19239 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:29.232558 19239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:29.232879 19239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:29.235762 19239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "1a46bec8d1564dd69d084c8393e7da1a"
format_stamp: "Formatted at 2026-06-20 03:10:29 on dist-test-slave-xv68"
server_key: "5aeb1d370854106287eb3e9be7ace0fe"
server_key_iv: "1eca7472cae2b36a04c7ea9146cd2b21"
server_key_version: "encryptionkey@0"
I20260620 03:10:29.236532 19239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1a46bec8d1564dd69d084c8393e7da1a"
format_stamp: "Formatted at 2026-06-20 03:10:29 on dist-test-slave-xv68"
server_key: "5aeb1d370854106287eb3e9be7ace0fe"
server_key_iv: "1eca7472cae2b36a04c7ea9146cd2b21"
server_key_version: "encryptionkey@0"
I20260620 03:10:29.241773 19239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:10:29.245560 19253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:29.246891 19239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:10:29.247090 19239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "1a46bec8d1564dd69d084c8393e7da1a"
format_stamp: "Formatted at 2026-06-20 03:10:29 on dist-test-slave-xv68"
server_key: "5aeb1d370854106287eb3e9be7ace0fe"
server_key_iv: "1eca7472cae2b36a04c7ea9146cd2b21"
server_key_version: "encryptionkey@0"
I20260620 03:10:29.247385 19239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:29.277163 19239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:29.277983 19239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:29.278213 19239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:29.279115 19239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:29.280566 19239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:29.280651 19239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:29.280736 19239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:29.280791 19239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:29.314466 19239 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32919
I20260620 03:10:29.314541 19365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32919 every 8 connection(s)
I20260620 03:10:29.315863 19239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:10:29.322516 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 19239
I20260620 03:10:29.322675 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:10:29.323109 388 external_mini_cluster.cc:1428] Setting key 70c1371d227e3a48adc114b1cd86cad4
I20260620 03:10:29.326071 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44551
--builtin_ntp_servers=127.0.97.20:37065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 03:10:29.329003 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:29.318337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44519 (local address 127.0.97.62:44551)
0620 03:10:29.318772 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:29.318778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:29.319884 (+ 1106us) server_negotiation.cc:407] Connection header received
0620 03:10:29.321691 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:29.321697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:29.321787 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:29.321902 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:29.323047 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:29.324022 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:29.326051 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:29.326547 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:29.326641 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:29.328283 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:29.328289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:29.328292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:29.328309 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:29.328350 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:29.328361 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:29.328429 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:29.328679 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 03:10:29.328754 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":243,"threads_started":1}
I20260620 03:10:29.330389 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:29.318823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44551 (local address 127.0.97.1:44519)
0620 03:10:29.319723 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:29.319757 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:29.321429 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:29.322148 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:29.322171 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:29.322255 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:10:29.322883 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:29.322902 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:29.324161 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:29.324165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:29.325871 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:29.325888 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:29.326751 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:29.326760 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:29.327130 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:29.327941 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:10:29.327947 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:10:29.327999 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:10:29.328099 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:10:29.328457 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:10:29.328470 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:10:29.328481 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 03:10:29.328667 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 03:10:29.329062 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":260,"threads_started":1}
I20260620 03:10:29.331889 19366 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44551
I20260620 03:10:29.332175 19366 heartbeater.cc:464] Registering TS with master...
I20260620 03:10:29.332890 19366 heartbeater.cc:511] Master 127.0.97.62:44551 requested a full tablet report, sending...
I20260620 03:10:29.335017 19180 ts_manager.cc:194] Registered new tserver with Master: 1a46bec8d1564dd69d084c8393e7da1a (127.0.97.1:32919)
I20260620 03:10:29.336571 19180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:44519
W20260620 03:10:29.652559 19370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:29.652815 19370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:29.652949 19370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:29.660063 19370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:29.660162 19370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:29.660342 19370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:10:29.669314 19370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:29.670678 19370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:29.672451 19370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:29.685257 19375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:29.685309 19376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:29.685222 19378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:29.688633 19370 server_base.cc:1034] running on GCE node
I20260620 03:10:29.689280 19370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:29.690182 19370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:29.691387 19370 hybrid_clock.cc:648] HybridClock initialized: now 1781925029691329 us; error 60 us; skew 500 ppm
I20260620 03:10:29.691689 19370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:10:29.693799 19370 webserver.cc:460] Webserver started at http://127.0.97.2:36593/ using document root <none> and password file <none>
I20260620 03:10:29.694449 19370 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:29.694576 19370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:29.694886 19370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:29.697659 19370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "0b69752c81f64022b9b0c75d176080e1"
format_stamp: "Formatted at 2026-06-20 03:10:29 on dist-test-slave-xv68"
server_key: "fcf10e276d38dcde907886ace980b0ef"
server_key_iv: "e352a48a0ec144f2c3b5daefae432da5"
server_key_version: "encryptionkey@0"
I20260620 03:10:29.698427 19370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0b69752c81f64022b9b0c75d176080e1"
format_stamp: "Formatted at 2026-06-20 03:10:29 on dist-test-slave-xv68"
server_key: "fcf10e276d38dcde907886ace980b0ef"
server_key_iv: "e352a48a0ec144f2c3b5daefae432da5"
server_key_version: "encryptionkey@0"
I20260620 03:10:29.703233 19370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 03:10:29.706465 19384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:29.707799 19370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:10:29.707966 19370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "0b69752c81f64022b9b0c75d176080e1"
format_stamp: "Formatted at 2026-06-20 03:10:29 on dist-test-slave-xv68"
server_key: "fcf10e276d38dcde907886ace980b0ef"
server_key_iv: "e352a48a0ec144f2c3b5daefae432da5"
server_key_version: "encryptionkey@0"
I20260620 03:10:29.708177 19370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:29.721796 19370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:29.722568 19370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:29.722838 19370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:29.723820 19370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:29.725234 19370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:29.725364 19370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:29.725488 19370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:29.725574 19370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:29.758600 19370 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42631
I20260620 03:10:29.758649 19496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42631 every 8 connection(s)
I20260620 03:10:29.759968 19370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:10:29.762938 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 19370
I20260620 03:10:29.763108 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:10:29.763736 388 external_mini_cluster.cc:1428] Setting key d6db240d4712f6f4ba52ac86c3aa9ac5
I20260620 03:10:29.766752 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44551
--builtin_ntp_servers=127.0.97.20:37065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 03:10:29.774478 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:29.762387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33123 (local address 127.0.97.62:44551)
0620 03:10:29.762608 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:29.762613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:29.764203 (+ 1590us) server_negotiation.cc:407] Connection header received
0620 03:10:29.766376 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:29.766381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:29.766443 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:29.766527 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:29.767936 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:29.768672 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:29.771180 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:29.771885 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:29.772018 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:29.773639 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:29.773647 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:29.773652 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:29.773674 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:29.773712 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:29.773727 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:29.773813 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:29.774056 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 03:10:29.774157 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 03:10:29.775729 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:29.762886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44551 (local address 127.0.97.2:33123)
0620 03:10:29.764014 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:29.764066 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:29.766080 (+ 2014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:29.766796 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:29.766807 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:29.766927 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:10:29.767747 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:29.767773 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:29.768832 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:29.768840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:29.771021 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:29.771038 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:29.772120 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:29.772129 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:29.772464 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:29.773264 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:10:29.773270 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:10:29.773330 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:10:29.773423 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:10:29.773834 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:10:29.773846 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:10:29.773857 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 03:10:29.774032 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 03:10:29.774424 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":397,"threads_started":1}
I20260620 03:10:29.777115 19497 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44551
I20260620 03:10:29.777401 19497 heartbeater.cc:464] Registering TS with master...
I20260620 03:10:29.778234 19497 heartbeater.cc:511] Master 127.0.97.62:44551 requested a full tablet report, sending...
I20260620 03:10:29.779667 19180 ts_manager.cc:194] Registered new tserver with Master: 0b69752c81f64022b9b0c75d176080e1 (127.0.97.2:42631)
I20260620 03:10:29.780679 19180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:33123
W20260620 03:10:30.090194 19500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:30.090446 19500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:30.090531 19500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:30.098120 19500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:30.098223 19500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:30.098330 19500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:10:30.107975 19500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:30.109159 19500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:30.110801 19500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:30.124121 19505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:30.124078 19506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:30.124125 19508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:30.125061 19500 server_base.cc:1034] running on GCE node
I20260620 03:10:30.125568 19500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:30.126497 19500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:30.127723 19500 hybrid_clock.cc:648] HybridClock initialized: now 1781925030127650 us; error 59 us; skew 500 ppm
I20260620 03:10:30.127981 19500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:10:30.130097 19500 webserver.cc:460] Webserver started at http://127.0.97.3:42381/ using document root <none> and password file <none>
I20260620 03:10:30.130693 19500 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:30.130777 19500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:30.131038 19500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:30.133739 19500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "d9ed9a49a56747bc85528ebe45e540bc"
format_stamp: "Formatted at 2026-06-20 03:10:30 on dist-test-slave-xv68"
server_key: "2560438e31398e1d583b1cd92793c8bd"
server_key_iv: "43630224e86634b5a06dcf1c26ac46e7"
server_key_version: "encryptionkey@0"
I20260620 03:10:30.134508 19500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d9ed9a49a56747bc85528ebe45e540bc"
format_stamp: "Formatted at 2026-06-20 03:10:30 on dist-test-slave-xv68"
server_key: "2560438e31398e1d583b1cd92793c8bd"
server_key_iv: "43630224e86634b5a06dcf1c26ac46e7"
server_key_version: "encryptionkey@0"
I20260620 03:10:30.139369 19500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 03:10:30.142644 19514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:30.144006 19500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 03:10:30.144174 19500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "d9ed9a49a56747bc85528ebe45e540bc"
format_stamp: "Formatted at 2026-06-20 03:10:30 on dist-test-slave-xv68"
server_key: "2560438e31398e1d583b1cd92793c8bd"
server_key_iv: "43630224e86634b5a06dcf1c26ac46e7"
server_key_version: "encryptionkey@0"
I20260620 03:10:30.144443 19500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:30.157905 19500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:30.158670 19500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:30.158946 19500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:30.159895 19500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:30.161303 19500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:30.161419 19500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:30.161553 19500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:30.161641 19500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:30.194265 19500 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37907
I20260620 03:10:30.194351 19626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37907 every 8 connection(s)
I20260620 03:10:30.195566 19500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:10:30.205502 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 19500
I20260620 03:10:30.205713 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:10:30.206226 388 external_mini_cluster.cc:1428] Setting key 0f4a69a41b13a437721136f30db9e297
I20260620 03:10:30.207599 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.197985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37591 (local address 127.0.97.62:44551)
0620 03:10:30.198213 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:30.198220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:30.199623 (+ 1403us) server_negotiation.cc:407] Connection header received
0620 03:10:30.201317 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:30.201322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:30.201386 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:30.201470 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:30.202422 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.203114 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.204617 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.204997 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.205086 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.206512 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:30.206517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:30.206520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:30.206538 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:30.206569 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:30.206581 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:30.206685 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:30.207150 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 03:10:30.207251 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 03:10:30.208340 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.198496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44551 (local address 127.0.97.3:37591)
0620 03:10:30.199472 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:30.199512 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:30.201067 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:30.201664 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:30.201671 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:30.201749 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:10:30.202283 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:30.202298 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.203249 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.203253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:30.204497 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:30.204508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.205202 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.205207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:30.205391 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.206163 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:10:30.206166 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:10:30.206203 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:10:30.206300 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:10:30.206703 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:10:30.206712 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:10:30.206718 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 03:10:30.206865 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 03:10:30.207124 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":313,"threads_started":1}
I20260620 03:10:30.209507 19627 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44551
I20260620 03:10:30.209789 19627 heartbeater.cc:464] Registering TS with master...
I20260620 03:10:30.210449 19627 heartbeater.cc:511] Master 127.0.97.62:44551 requested a full tablet report, sending...
I20260620 03:10:30.211853 19180 ts_manager.cc:194] Registered new tserver with Master: d9ed9a49a56747bc85528ebe45e540bc (127.0.97.3:37907)
I20260620 03:10:30.212726 19180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:37591
I20260620 03:10:30.221331 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:10:30.231169 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.225318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51336 (local address 127.0.97.62:44551)
0620 03:10:30.225504 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:30.225513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:30.225834 (+ 321us) server_negotiation.cc:407] Connection header received
0620 03:10:30.226023 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:30.226030 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:30.226100 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:30.226207 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:30.226699 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.227476 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.228819 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.229207 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.229339 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.230434 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:30.230440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:30.230443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:30.230460 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:30.230488 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:30.230500 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:30.230647 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:30.230866 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 03:10:30.230928 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 03:10:30.234826 19180 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:10:30.241592 19180 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:10:30.266603 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.253345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32919 (local address 127.0.0.1:52892)
0620 03:10:30.254614 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:30.254666 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:30.254842 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:30.256074 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:30.256085 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:30.256125 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:30.256636 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:30.256652 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.259165 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.259173 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:30.262443 (+ 3270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:30.262463 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.265260 (+ 2797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.265269 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:30.265582 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.265650 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 03:10:30.265838 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 03:10:30.266101 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"thread_start_us":851,"threads_started":1}
I20260620 03:10:30.267915 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.253591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.0.97.1:32919)
0620 03:10:30.254181 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:30.254192 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:30.255068 (+ 876us) server_negotiation.cc:407] Connection header received
0620 03:10:30.255198 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:30.255209 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:30.255535 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:30.255747 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:30.256866 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.258960 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.262694 (+ 3734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.264642 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.264800 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.265104 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:30.267186 (+ 2082us) server_negotiation.cc:299] Negotiation successful
0620 03:10:30.267447 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":331,"threads_started":1}
I20260620 03:10:30.274606 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.254006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42631 (local address 127.0.0.1:40562)
0620 03:10:30.268431 (+ 14425us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:30.268456 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:30.268583 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:30.269785 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:30.269790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:30.269823 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:30.270010 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:30.270023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.271510 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.271515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:30.272668 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:30.272678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.273856 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.273862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:30.274059 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.274095 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 03:10:30.274186 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 03:10:30.274340 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14253,"threads_started":1}
I20260620 03:10:30.275705 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.254661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40562 (local address 127.0.97.2:42631)
0620 03:10:30.269307 (+ 14646us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:30.269313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:30.269337 (+ 24us) server_negotiation.cc:407] Connection header received
0620 03:10:30.269415 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:30.269421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:30.269592 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:30.269796 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:30.270156 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.271366 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.272821 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.273694 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.273837 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.274092 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:30.275152 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0620 03:10:30.275328 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14488,"thread_start_us":14358,"threads_started":1}
I20260620 03:10:30.279309 19301 tablet_service.cc:1467] Processing CreateTablet for tablet 282117407c804001aca3108d74070f56 (DEFAULT_TABLE table=test-table [id=9bac41dbf5a64f5981c8fbe9361e2689]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:10:30.280435 19432 tablet_service.cc:1467] Processing CreateTablet for tablet 282117407c804001aca3108d74070f56 (DEFAULT_TABLE table=test-table [id=9bac41dbf5a64f5981c8fbe9361e2689]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:10:30.281260 19301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 282117407c804001aca3108d74070f56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:30.281730 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.255099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37907 (local address 127.0.0.1:45494)
0620 03:10:30.274834 (+ 19735us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:30.274857 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:30.274973 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:30.276036 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:30.276041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:30.276067 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:30.276201 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:30.276207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.278041 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.278049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:30.279233 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:30.279246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.280984 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.280990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:30.281167 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.281209 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 03:10:30.281308 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 03:10:30.281436 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19605,"thread_start_us":19836,"threads_started":1}
I20260620 03:10:30.281783 19432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 282117407c804001aca3108d74070f56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:30.281807 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.255744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45494 (local address 127.0.97.3:37907)
0620 03:10:30.274608 (+ 18864us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:30.274618 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:30.275108 (+ 490us) server_negotiation.cc:407] Connection header received
0620 03:10:30.275209 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:30.275216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:30.275482 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:30.275672 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:30.276351 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.277922 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.279481 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.280829 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.280974 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.281232 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:30.281357 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 03:10:30.281500 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18693,"thread_start_us":18561,"threads_started":1}
I20260620 03:10:30.290304 19562 tablet_service.cc:1467] Processing CreateTablet for tablet 282117407c804001aca3108d74070f56 (DEFAULT_TABLE table=test-table [id=9bac41dbf5a64f5981c8fbe9361e2689]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:10:30.291457 19562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 282117407c804001aca3108d74070f56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:30.296263 19646 tablet_bootstrap.cc:492] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1: Bootstrap starting.
I20260620 03:10:30.297021 19645 tablet_bootstrap.cc:492] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a: Bootstrap starting.
I20260620 03:10:30.300771 19645 tablet_bootstrap.cc:654] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:30.301800 19646 tablet_bootstrap.cc:654] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:30.302019 19645 log.cc:824] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:30.302906 19646 log.cc:824] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:30.304805 19649 tablet_bootstrap.cc:492] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc: Bootstrap starting.
I20260620 03:10:30.305545 19645 tablet_bootstrap.cc:492] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a: No bootstrap required, opened a new log
I20260620 03:10:30.305850 19645 ts_tablet_manager.cc:1397] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260620 03:10:30.306397 19646 tablet_bootstrap.cc:492] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1: No bootstrap required, opened a new log
I20260620 03:10:30.306690 19646 ts_tablet_manager.cc:1397] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260620 03:10:30.308277 19649 tablet_bootstrap.cc:654] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:30.309422 19649 log.cc:824] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:30.312314 19649 tablet_bootstrap.cc:492] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc: No bootstrap required, opened a new log
I20260620 03:10:30.312556 19649 ts_tablet_manager.cc:1397] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 03:10:30.314098 19645 raft_consensus.cc:348] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a46bec8d1564dd69d084c8393e7da1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32919 } } peers { permanent_uuid: "d9ed9a49a56747bc85528ebe45e540bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37907 } } peers { permanent_uuid: "0b69752c81f64022b9b0c75d176080e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42631 } }
I20260620 03:10:30.314657 19645 raft_consensus.cc:374] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:30.314821 19645 raft_consensus.cc:729] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a46bec8d1564dd69d084c8393e7da1a, State: Initialized, Role: FOLLOWER
I20260620 03:10:30.315706 19645 consensus_queue.cc:260] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a46bec8d1564dd69d084c8393e7da1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32919 } } peers { permanent_uuid: "d9ed9a49a56747bc85528ebe45e540bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37907 } } peers { permanent_uuid: "0b69752c81f64022b9b0c75d176080e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42631 } }
I20260620 03:10:30.316968 19646 raft_consensus.cc:348] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a46bec8d1564dd69d084c8393e7da1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32919 } } peers { permanent_uuid: "d9ed9a49a56747bc85528ebe45e540bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37907 } } peers { permanent_uuid: "0b69752c81f64022b9b0c75d176080e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42631 } }
I20260620 03:10:30.317420 19646 raft_consensus.cc:374] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:30.317549 19646 raft_consensus.cc:729] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b69752c81f64022b9b0c75d176080e1, State: Initialized, Role: FOLLOWER
I20260620 03:10:30.319167 19646 consensus_queue.cc:260] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a46bec8d1564dd69d084c8393e7da1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32919 } } peers { permanent_uuid: "d9ed9a49a56747bc85528ebe45e540bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37907 } } peers { permanent_uuid: "0b69752c81f64022b9b0c75d176080e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42631 } }
I20260620 03:10:30.323870 19645 ts_tablet_manager.cc:1428] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a: Time spent starting tablet: real 0.018s user 0.009s sys 0.007s
I20260620 03:10:30.323817 19649 raft_consensus.cc:348] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a46bec8d1564dd69d084c8393e7da1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32919 } } peers { permanent_uuid: "d9ed9a49a56747bc85528ebe45e540bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37907 } } peers { permanent_uuid: "0b69752c81f64022b9b0c75d176080e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42631 } }
I20260620 03:10:30.324404 19366 heartbeater.cc:503] Master 127.0.97.62:44551 was elected leader, sending a full tablet report...
I20260620 03:10:30.324388 19649 raft_consensus.cc:374] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:30.324520 19649 raft_consensus.cc:729] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ed9a49a56747bc85528ebe45e540bc, State: Initialized, Role: FOLLOWER
I20260620 03:10:30.325317 19649 consensus_queue.cc:260] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a46bec8d1564dd69d084c8393e7da1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32919 } } peers { permanent_uuid: "d9ed9a49a56747bc85528ebe45e540bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37907 } } peers { permanent_uuid: "0b69752c81f64022b9b0c75d176080e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42631 } }
I20260620 03:10:30.328203 19497 heartbeater.cc:503] Master 127.0.97.62:44551 was elected leader, sending a full tablet report...
I20260620 03:10:30.328200 19646 ts_tablet_manager.cc:1428] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1: Time spent starting tablet: real 0.021s user 0.016s sys 0.002s
I20260620 03:10:30.332530 19627 heartbeater.cc:503] Master 127.0.97.62:44551 was elected leader, sending a full tablet report...
I20260620 03:10:30.333505 19649 ts_tablet_manager.cc:1428] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc: Time spent starting tablet: real 0.021s user 0.020s sys 0.000s
I20260620 03:10:30.368427 19652 raft_consensus.cc:482] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:10:30.368727 19652 raft_consensus.cc:504] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a46bec8d1564dd69d084c8393e7da1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32919 } } peers { permanent_uuid: "d9ed9a49a56747bc85528ebe45e540bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37907 } } peers { permanent_uuid: "0b69752c81f64022b9b0c75d176080e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42631 } }
I20260620 03:10:30.369603 19653 raft_consensus.cc:482] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:10:30.369891 19653 raft_consensus.cc:504] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a46bec8d1564dd69d084c8393e7da1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32919 } } peers { permanent_uuid: "d9ed9a49a56747bc85528ebe45e540bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37907 } } peers { permanent_uuid: "0b69752c81f64022b9b0c75d176080e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42631 } }
I20260620 03:10:30.370714 19652 leader_election.cc:288] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a46bec8d1564dd69d084c8393e7da1a (127.0.97.1:32919), d9ed9a49a56747bc85528ebe45e540bc (127.0.97.3:37907)
I20260620 03:10:30.371920 19653 leader_election.cc:288] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a46bec8d1564dd69d084c8393e7da1a (127.0.97.1:32919), 0b69752c81f64022b9b0c75d176080e1 (127.0.97.2:42631)
I20260620 03:10:30.382885 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.371016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52073 (local address 127.0.97.1:32919)
0620 03:10:30.371239 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:30.371245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:30.371790 (+ 545us) server_negotiation.cc:407] Connection header received
0620 03:10:30.372915 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:30.372925 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:30.373051 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:30.373130 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:30.374140 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.375618 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.379787 (+ 4169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.381633 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.381752 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.381864 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:30.382437 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 03:10:30.382569 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":70}
I20260620 03:10:30.383765 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.370955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32919 (local address 127.0.97.2:52073)
0620 03:10:30.371691 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:30.371713 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:30.372690 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:30.373375 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:30.373383 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:30.373420 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:30.373943 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:30.373957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.375852 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.375859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:30.378772 (+ 2913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:30.378792 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.381997 (+ 3205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.382003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:30.382249 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.382293 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 03:10:30.383346 (+ 1053us) client_negotiation.cc:240] Negotiation successful
0620 03:10:30.383499 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":1,"thread_start_us":380,"threads_started":1}
I20260620 03:10:30.384784 19321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282117407c804001aca3108d74070f56" candidate_uuid: "0b69752c81f64022b9b0c75d176080e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a46bec8d1564dd69d084c8393e7da1a" is_pre_election: true
I20260620 03:10:30.385363 19321 raft_consensus.cc:2454] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b69752c81f64022b9b0c75d176080e1 in term 0.
I20260620 03:10:30.386183 19388 leader_election.cc:302] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b69752c81f64022b9b0c75d176080e1, 1a46bec8d1564dd69d084c8393e7da1a; no voters:
I20260620 03:10:30.386615 19652 raft_consensus.cc:2781] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:10:30.386780 19652 raft_consensus.cc:482] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:10:30.386895 19652 raft_consensus.cc:3037] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:30.387678 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.372329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32919 (local address 127.0.97.3:34751)
0620 03:10:30.373083 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:30.373105 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:30.373241 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:30.380184 (+ 6943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:30.380190 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:30.380219 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:30.380392 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:30.380404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.382747 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.382752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:30.385181 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:30.385196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.386826 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.386832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:30.387053 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.387089 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 03:10:30.387160 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 03:10:30.387348 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":182}
I20260620 03:10:30.388381 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.373473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52609 (local address 127.0.97.2:42631)
0620 03:10:30.373723 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:30.373732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:30.379636 (+ 5904us) server_negotiation.cc:407] Connection header received
0620 03:10:30.380726 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:30.380731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:30.380806 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:30.380869 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:30.381607 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.382791 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.385794 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.387200 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.387329 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.387427 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:30.387925 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 03:10:30.388064 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":73}
I20260620 03:10:30.388630 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.372411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34751 (local address 127.0.97.1:32919)
0620 03:10:30.379828 (+ 7417us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:30.379834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:30.379856 (+ 22us) server_negotiation.cc:407] Connection header received
0620 03:10:30.379916 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:30.379920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:30.379991 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:30.380538 (+ 547us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:30.380858 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.382597 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.385358 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.386632 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.387993 (+ 1361us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.388119 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:30.388170 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 03:10:30.388295 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":7260,"thread_start_us":7932,"threads_started":1}
I20260620 03:10:30.389012 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.372597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42631 (local address 127.0.97.3:52609)
0620 03:10:30.379518 (+ 6921us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:30.379546 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:30.380577 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:30.381058 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:30.381062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:30.381089 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:30.381425 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:30.381436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.382975 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.382980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:30.385579 (+ 2599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:30.385593 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.387574 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.387580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:30.387749 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.387778 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 03:10:30.388584 (+ 806us) client_negotiation.cc:240] Negotiation successful
0620 03:10:30.388718 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6761,"thread_start_us":6530,"threads_started":1}
I20260620 03:10:30.389142 19321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282117407c804001aca3108d74070f56" candidate_uuid: "d9ed9a49a56747bc85528ebe45e540bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a46bec8d1564dd69d084c8393e7da1a" is_pre_election: true
I20260620 03:10:30.389353 19321 raft_consensus.cc:2454] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9ed9a49a56747bc85528ebe45e540bc in term 0.
I20260620 03:10:30.389530 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.371849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53765 (local address 127.0.97.3:37907)
0620 03:10:30.372051 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:30.372057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:30.382096 (+ 10039us) server_negotiation.cc:407] Connection header received
0620 03:10:30.382340 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:30.382348 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:30.382423 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:30.382479 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:30.383240 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.384758 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.387189 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.388413 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.388505 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.388604 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:30.389180 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 03:10:30.389306 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":66}
I20260620 03:10:30.389886 19452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282117407c804001aca3108d74070f56" candidate_uuid: "d9ed9a49a56747bc85528ebe45e540bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b69752c81f64022b9b0c75d176080e1" is_pre_election: true
I20260620 03:10:30.390409 19652 raft_consensus.cc:504] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a46bec8d1564dd69d084c8393e7da1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32919 } } peers { permanent_uuid: "d9ed9a49a56747bc85528ebe45e540bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37907 } } peers { permanent_uuid: "0b69752c81f64022b9b0c75d176080e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42631 } }
I20260620 03:10:30.390564 19518 leader_election.cc:302] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a46bec8d1564dd69d084c8393e7da1a, d9ed9a49a56747bc85528ebe45e540bc; no voters:
I20260620 03:10:30.390838 19452 raft_consensus.cc:2379] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9ed9a49a56747bc85528ebe45e540bc in current term 1: Already voted for candidate 0b69752c81f64022b9b0c75d176080e1 in this term.
I20260620 03:10:30.390983 19653 raft_consensus.cc:2781] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:10:30.391136 19653 raft_consensus.cc:482] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:10:30.391172 19652 leader_election.cc:288] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [CANDIDATE]: Term 1 election: Requested vote from peers 1a46bec8d1564dd69d084c8393e7da1a (127.0.97.1:32919), d9ed9a49a56747bc85528ebe45e540bc (127.0.97.3:37907)
I20260620 03:10:30.391258 19653 raft_consensus.cc:3037] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:30.391383 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.371762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37907 (local address 127.0.97.2:53765)
0620 03:10:30.382003 (+ 10241us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:30.382021 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:30.382196 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:30.382636 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:30.382643 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:30.382679 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:30.383046 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:30.383060 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.384934 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.384939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:30.386962 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:30.386979 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.388766 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.388773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:30.388972 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.389019 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 03:10:30.389729 (+ 710us) client_negotiation.cc:240] Negotiation successful
0620 03:10:30.389897 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10100,"mutex_wait_us":1,"thread_start_us":9905,"threads_started":1}
I20260620 03:10:30.391556 19321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282117407c804001aca3108d74070f56" candidate_uuid: "0b69752c81f64022b9b0c75d176080e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a46bec8d1564dd69d084c8393e7da1a"
I20260620 03:10:30.391757 19321 raft_consensus.cc:3037] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:30.392493 19582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282117407c804001aca3108d74070f56" candidate_uuid: "0b69752c81f64022b9b0c75d176080e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9ed9a49a56747bc85528ebe45e540bc" is_pre_election: true
I20260620 03:10:30.393260 19581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282117407c804001aca3108d74070f56" candidate_uuid: "0b69752c81f64022b9b0c75d176080e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9ed9a49a56747bc85528ebe45e540bc"
I20260620 03:10:30.394816 19321 raft_consensus.cc:2454] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b69752c81f64022b9b0c75d176080e1 in term 1.
I20260620 03:10:30.394763 19653 raft_consensus.cc:504] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a46bec8d1564dd69d084c8393e7da1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32919 } } peers { permanent_uuid: "d9ed9a49a56747bc85528ebe45e540bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37907 } } peers { permanent_uuid: "0b69752c81f64022b9b0c75d176080e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42631 } }
I20260620 03:10:30.395251 19582 raft_consensus.cc:2379] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0b69752c81f64022b9b0c75d176080e1 in current term 1: Already voted for candidate d9ed9a49a56747bc85528ebe45e540bc in this term.
I20260620 03:10:30.395222 19388 leader_election.cc:302] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b69752c81f64022b9b0c75d176080e1, 1a46bec8d1564dd69d084c8393e7da1a; no voters: d9ed9a49a56747bc85528ebe45e540bc
I20260620 03:10:30.395628 19653 leader_election.cc:288] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [CANDIDATE]: Term 1 election: Requested vote from peers 1a46bec8d1564dd69d084c8393e7da1a (127.0.97.1:32919), 0b69752c81f64022b9b0c75d176080e1 (127.0.97.2:42631)
I20260620 03:10:30.395682 19652 raft_consensus.cc:2781] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:30.395989 19452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282117407c804001aca3108d74070f56" candidate_uuid: "d9ed9a49a56747bc85528ebe45e540bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b69752c81f64022b9b0c75d176080e1"
I20260620 03:10:30.396227 19652 raft_consensus.cc:686] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [term 1 LEADER]: Becoming Leader. State: Replica: 0b69752c81f64022b9b0c75d176080e1, State: Running, Role: LEADER
I20260620 03:10:30.396869 19321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282117407c804001aca3108d74070f56" candidate_uuid: "d9ed9a49a56747bc85528ebe45e540bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a46bec8d1564dd69d084c8393e7da1a"
I20260620 03:10:30.396746 19652 consensus_queue.cc:237] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a46bec8d1564dd69d084c8393e7da1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32919 } } peers { permanent_uuid: "d9ed9a49a56747bc85528ebe45e540bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37907 } } peers { permanent_uuid: "0b69752c81f64022b9b0c75d176080e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42631 } }
I20260620 03:10:30.397138 19321 raft_consensus.cc:2379] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d9ed9a49a56747bc85528ebe45e540bc in current term 1: Already voted for candidate 0b69752c81f64022b9b0c75d176080e1 in this term.
I20260620 03:10:30.400400 19515 leader_election.cc:302] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9ed9a49a56747bc85528ebe45e540bc; no voters: 0b69752c81f64022b9b0c75d176080e1, 1a46bec8d1564dd69d084c8393e7da1a
I20260620 03:10:30.400662 19653 raft_consensus.cc:2726] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 03:10:30.401830 19178 catalog_manager.cc:5495] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b69752c81f64022b9b0c75d176080e1 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0b69752c81f64022b9b0c75d176080e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a46bec8d1564dd69d084c8393e7da1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9ed9a49a56747bc85528ebe45e540bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b69752c81f64022b9b0c75d176080e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42631 } health_report { overall_health: HEALTHY } } }
I20260620 03:10:30.445828 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.440771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40578 (local address 127.0.97.2:42631)
0620 03:10:30.440947 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:30.440951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:30.441029 (+ 78us) server_negotiation.cc:407] Connection header received
0620 03:10:30.441205 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:30.441209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:30.441282 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:30.441406 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:10:30.441729 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.442458 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.443851 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.444243 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.444341 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.444837 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:10:30.445298 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:10:30.445421 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:30.445562 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 03:10:30.445597 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 03:10:30.447202 19628 tablet.cc:2369] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:10:30.453759 19582 raft_consensus.cc:1261] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [term 1 FOLLOWER]: Refusing update from remote peer 0b69752c81f64022b9b0c75d176080e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:10:30.453799 19321 raft_consensus.cc:1261] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a [term 1 FOLLOWER]: Refusing update from remote peer 0b69752c81f64022b9b0c75d176080e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:10:30.454574 19658 consensus_queue.cc:1035] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9ed9a49a56747bc85528ebe45e540bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:10:30.454929 19652 consensus_queue.cc:1035] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a46bec8d1564dd69d084c8393e7da1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:10:30.465179 19661 mvcc.cc:205] Tried to move back new op lower bound from 7298764924730130432 to 7298764924509696000. Current Snapshot: MvccSnapshot[applied={T|T < 7298764924730130432}]
I20260620 03:10:30.465655 19663 mvcc.cc:205] Tried to move back new op lower bound from 7298764924730130432 to 7298764924509696000. Current Snapshot: MvccSnapshot[applied={T|T < 7298764924730130432}]
I20260620 03:10:30.469321 19662 mvcc.cc:205] Tried to move back new op lower bound from 7298764924730130432 to 7298764924509696000. Current Snapshot: MvccSnapshot[applied={T|T < 7298764924730130432}]
I20260620 03:10:30.481388 19180 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:10:30.481644 19180 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:10:30.484920 19180 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb8d749f069745209efcb8407d8df240: Sending DeleteTablet for 3 replicas of tablet 282117407c804001aca3108d74070f56
I20260620 03:10:30.486039 19562 tablet_service.cc:1514] Processing DeleteTablet for tablet 282117407c804001aca3108d74070f56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:10:30 UTC) from {username='slave'} at 127.0.0.1:45494
I20260620 03:10:30.486117 19432 tablet_service.cc:1514] Processing DeleteTablet for tablet 282117407c804001aca3108d74070f56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:10:30 UTC) from {username='slave'} at 127.0.0.1:40562
I20260620 03:10:30.486322 19301 tablet_service.cc:1514] Processing DeleteTablet for tablet 282117407c804001aca3108d74070f56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:10:30 UTC) from {username='slave'} at 127.0.0.1:52892
I20260620 03:10:30.487982 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 19239
I20260620 03:10:30.504408 19671 tablet_replica.cc:330] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc: stopping tablet replica
I20260620 03:10:30.505012 19672 tablet_replica.cc:330] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a: stopping tablet replica
I20260620 03:10:30.505429 19672 raft_consensus.cc:2229] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:10:30.505954 19672 raft_consensus.cc:2258] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:10:30.506667 19671 raft_consensus.cc:2229] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:10:30.507155 19671 raft_consensus.cc:2258] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:10:30.508926 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.489465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.0.97.1:32919)
0620 03:10:30.489631 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:30.489639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:30.494917 (+ 5278us) server_negotiation.cc:407] Connection header received
0620 03:10:30.495823 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:30.495832 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:30.495991 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:30.496085 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:30.497441 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.498982 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.503956 (+ 4974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.504685 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.504815 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.506531 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:30.506541 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:30.506559 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:30.507344 (+ 785us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:30.507403 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:30.507430 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:30.507548 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:30.508497 (+ 949us) server_negotiation.cc:299] Negotiation successful
0620 03:10:30.508611 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 03:10:30.509334 19671 ts_tablet_manager.cc:1905] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:10:30.510123 19361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:30.516997 19670 tablet_replica.cc:330] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1: stopping tablet replica
I20260620 03:10:30.517474 19670 raft_consensus.cc:2229] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:10:30.518267 19670 raft_consensus.cc:2258] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:10:30.520358 19670 ts_tablet_manager.cc:1905] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:10:30.523855 19671 ts_tablet_manager.cc:1918] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:10:30.524061 19671 log.cc:1196] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/282117407c804001aca3108d74070f56
I20260620 03:10:30.524649 19671 ts_tablet_manager.cc:1939] T 282117407c804001aca3108d74070f56 P d9ed9a49a56747bc85528ebe45e540bc: Deleting consensus metadata
I20260620 03:10:30.526571 19167 catalog_manager.cc:4841] TS d9ed9a49a56747bc85528ebe45e540bc (127.0.97.3:37907): tablet 282117407c804001aca3108d74070f56 (table test-table [id=9bac41dbf5a64f5981c8fbe9361e2689]) successfully deleted
I20260620 03:10:30.528213 19670 ts_tablet_manager.cc:1918] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:10:30.528395 19670 log.cc:1196] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/282117407c804001aca3108d74070f56
I20260620 03:10:30.529011 19670 ts_tablet_manager.cc:1939] T 282117407c804001aca3108d74070f56 P 0b69752c81f64022b9b0c75d176080e1: Deleting consensus metadata
I20260620 03:10:30.530910 19164 catalog_manager.cc:4841] TS 0b69752c81f64022b9b0c75d176080e1 (127.0.97.2:42631): tablet 282117407c804001aca3108d74070f56 (table test-table [id=9bac41dbf5a64f5981c8fbe9361e2689]) successfully deleted
I20260620 03:10:30.516359 19672 ts_tablet_manager.cc:1905] T 282117407c804001aca3108d74070f56 P 1a46bec8d1564dd69d084c8393e7da1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:10:30.610097 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 19239
I20260620 03:10:30.628350 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 19370
W20260620 03:10:30.628597 19167 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:32919 (error 108)
I20260620 03:10:30.630501 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.630020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32919 (local address 127.0.0.1:52922)
0620 03:10:30.630188 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:30.630325 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 03:10:30.631862 19167 catalog_manager.cc:4584] TS 1a46bec8d1564dd69d084c8393e7da1a (127.0.97.1:32919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 282117407c804001aca3108d74070f56: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32919: connect: Connection refused (error 111)
I20260620 03:10:30.636217 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.629624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40594 (local address 127.0.97.2:42631)
0620 03:10:30.629798 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:30.629812 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:30.629833 (+ 21us) server_negotiation.cc:407] Connection header received
0620 03:10:30.629900 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:30.629906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:30.630030 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:30.630131 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:30.630733 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.631568 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.632965 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.633424 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.633557 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.634630 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:30.634639 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:30.634652 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:30.635236 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:30.635396 (+ 160us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:30.635422 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:30.635536 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:30.635768 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 03:10:30.635876 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 03:10:30.636977 19492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:30.685741 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.685110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32919 (local address 127.0.0.1:52928)
0620 03:10:30.685486 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:30.685581 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 03:10:30.686829 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.686471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32919 (local address 127.0.0.1:52942)
0620 03:10:30.686625 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:30.686721 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 03:10:30.747551 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.746957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32919 (local address 127.0.0.1:52952)
0620 03:10:30.747177 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:30.747311 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 03:10:30.747807 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 19370
I20260620 03:10:30.749265 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.748817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32919 (local address 127.0.0.1:52956)
0620 03:10:30.748998 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:30.749105 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 03:10:30.766244 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 19500
I20260620 03:10:30.772969 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.766974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.0.97.3:37907)
0620 03:10:30.767131 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:30.767135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:30.767152 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:10:30.767385 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:30.767389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:30.767485 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:30.767570 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:30.768032 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.768727 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.770000 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:30.770369 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:30.770499 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:30.771663 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:30.771672 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:30.771687 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:30.772258 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:30.772300 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:30.772320 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:30.772447 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:30.772574 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 03:10:30.772677 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 03:10:30.773804 19622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:30.829855 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.829370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32919 (local address 127.0.0.1:52960)
0620 03:10:30.829574 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:30.829678 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 03:10:30.830924 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:30.830544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32919 (local address 127.0.0.1:52970)
0620 03:10:30.830758 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:30.830829 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260620 03:10:30.869779 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 19500
I20260620 03:10:30.887225 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 19149
I20260620 03:10:30.888367 19210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:30.996143 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 19149
2026-06-20T03:10:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2717 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:10:31 dist-test-slave-xv68 krb5kdc[19681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:10:31 dist-test-slave-xv68 krb5kdc[19681](info): set up 2 sockets
Jun 20 03:10:31 dist-test-slave-xv68 krb5kdc[19681](info): commencing operation
krb5kdc: starting...
W20260620 03:10:32.099545 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:10:32 dist-test-slave-xv68 krb5kdc[19681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:10:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:10:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:10:32.344530 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44995
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:32.670053 19694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:32.670379 19694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:32.670534 19694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:32.677745 19694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:10:32.677896 19694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:32.678001 19694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:32.678113 19694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:10:32.678200 19694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:10:32.687690 19694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:32.688958 19694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:32.690241 19694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:32.701656 19699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:32.701803 19700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:32.701776 19702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:32.702546 19694 server_base.cc:1034] running on GCE node
I20260620 03:10:32.703338 19694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:32.704731 19694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:32.705974 19694 hybrid_clock.cc:648] HybridClock initialized: now 1781925032705932 us; error 52 us; skew 500 ppm
I20260620 03:10:32.706327 19694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:32 dist-test-slave-xv68 krb5kdc[19681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:32.714217 19694 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:10:32.715797 19694 webserver.cc:460] Webserver started at http://127.0.97.62:45479/ using document root <none> and password file <none>
I20260620 03:10:32.716496 19694 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:32.716641 19694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:32.716964 19694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:32.720023 19694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "a69eb360175640a6bd74282652cdeb2e"
format_stamp: "Formatted at 2026-06-20 03:10:32 on dist-test-slave-xv68"
server_key: "b6bca3ef4a68a76ce81526bdb9ab89e2"
server_key_iv: "9967250de63d2a17561e34aa035d38fd"
server_key_version: "encryptionkey@0"
I20260620 03:10:32.720770 19694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "a69eb360175640a6bd74282652cdeb2e"
format_stamp: "Formatted at 2026-06-20 03:10:32 on dist-test-slave-xv68"
server_key: "b6bca3ef4a68a76ce81526bdb9ab89e2"
server_key_iv: "9967250de63d2a17561e34aa035d38fd"
server_key_version: "encryptionkey@0"
I20260620 03:10:32.725945 19694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 03:10:32.729333 19709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:32.730643 19694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:10:32.730871 19694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "a69eb360175640a6bd74282652cdeb2e"
format_stamp: "Formatted at 2026-06-20 03:10:32 on dist-test-slave-xv68"
server_key: "b6bca3ef4a68a76ce81526bdb9ab89e2"
server_key_iv: "9967250de63d2a17561e34aa035d38fd"
server_key_version: "encryptionkey@0"
I20260620 03:10:32.731076 19694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:32.743265 19694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:32.747146 19694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:32.747507 19694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:32.765810 19694 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44995
I20260620 03:10:32.765832 19760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44995 every 8 connection(s)
I20260620 03:10:32.767231 19694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:10:32.767972 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 19694
I20260620 03:10:32.768154 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:10:32.768718 388 external_mini_cluster.cc:1428] Setting key 9c9689c560428d46c23f0c979381a3c8
I20260620 03:10:32.771778 19761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 03:10:32 dist-test-slave-xv68 krb5kdc[19681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:10:32.782131 19761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e: Bootstrap starting.
I20260620 03:10:32.785449 19761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:32.786468 19761 log.cc:824] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:32.789397 19761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e: No bootstrap required, opened a new log
I20260620 03:10:32.789716 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:32.770544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50378 (local address 127.0.97.62:44995)
0620 03:10:32.771348 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:32.771373 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:32.771414 (+ 41us) server_negotiation.cc:407] Connection header received
0620 03:10:32.773091 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:32.773113 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:32.773460 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:32.773829 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:32.775200 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:32.776603 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:32.777895 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:32.778613 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:32.778809 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:32.782731 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:32.782738 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:32.782751 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:32.782785 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:32.785779 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:32.786444 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:32.786460 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:32.786465 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:32.786652 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:32.787038 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:32.787043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:32.787047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:32.787617 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:32.787935 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:32.788349 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 03:10:32.788746 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":340,"threads_started":1}
I20260620 03:10:32.797787 19761 raft_consensus.cc:348] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69eb360175640a6bd74282652cdeb2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44995 } }
I20260620 03:10:32.798309 19761 raft_consensus.cc:374] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:32.798417 19761 raft_consensus.cc:729] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69eb360175640a6bd74282652cdeb2e, State: Initialized, Role: FOLLOWER
I20260620 03:10:32.798959 19761 consensus_queue.cc:260] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69eb360175640a6bd74282652cdeb2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44995 } }
I20260620 03:10:32.799211 19761 raft_consensus.cc:388] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:10:32.799393 19761 raft_consensus.cc:482] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:10:32.799576 19761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:32.801738 19761 raft_consensus.cc:504] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69eb360175640a6bd74282652cdeb2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44995 } }
I20260620 03:10:32.802258 19761 leader_election.cc:302] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a69eb360175640a6bd74282652cdeb2e; no voters:
I20260620 03:10:32.802829 19761 leader_election.cc:288] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:10:32.802968 19766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:32.803699 19766 raft_consensus.cc:686] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e [term 1 LEADER]: Becoming Leader. State: Replica: a69eb360175640a6bd74282652cdeb2e, State: Running, Role: LEADER
I20260620 03:10:32.804327 19766 consensus_queue.cc:237] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69eb360175640a6bd74282652cdeb2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44995 } }
I20260620 03:10:32.805035 19761 sys_catalog.cc:564] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:10:32.807498 19767 sys_catalog.cc:455] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a69eb360175640a6bd74282652cdeb2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69eb360175640a6bd74282652cdeb2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44995 } } }
I20260620 03:10:32.807852 19767 sys_catalog.cc:458] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:32.807123 19768 sys_catalog.cc:455] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a69eb360175640a6bd74282652cdeb2e. Latest consensus state: current_term: 1 leader_uuid: "a69eb360175640a6bd74282652cdeb2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69eb360175640a6bd74282652cdeb2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44995 } } }
I20260620 03:10:32.808355 19768 sys_catalog.cc:458] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:32.810168 19774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:10:32.815675 19774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:10:32.823848 19774 catalog_manager.cc:1347] Generated new cluster ID: 5e3477ac96c04327855f784b35909b96
I20260620 03:10:32.823961 19774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:10:32.843856 19774 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:10:32.845206 19774 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:10:32.853135 19774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e: Generated new TSK 0
I20260620 03:10:32.853935 19774 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:10:32.914654 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44995
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:33.240233 19788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:33.240509 19788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:33.240687 19788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:33.248422 19788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:33.248565 19788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:33.248760 19788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:10:33.258709 19788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:33.260026 19788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:33.261358 19788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:33.274518 19793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:33.274528 19796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:33.274544 19794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:33.275571 19788 server_base.cc:1034] running on GCE node
I20260620 03:10:33.276177 19788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:33.277138 19788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:33.278342 19788 hybrid_clock.cc:648] HybridClock initialized: now 1781925033278300 us; error 37 us; skew 500 ppm
I20260620 03:10:33.278661 19788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:33 dist-test-slave-xv68 krb5kdc[19681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:33.281920 19788 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:10:33.283454 19788 webserver.cc:460] Webserver started at http://127.0.97.1:38607/ using document root <none> and password file <none>
I20260620 03:10:33.284138 19788 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:33.284269 19788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:33.284587 19788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:33.287496 19788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "5e6dcb476a014f398abfacfc165b5507"
format_stamp: "Formatted at 2026-06-20 03:10:33 on dist-test-slave-xv68"
server_key: "a952344c3ca8daf4773a376acba0da2f"
server_key_iv: "69da616ecfba28b01ec0a5e0b517eaac"
server_key_version: "encryptionkey@0"
I20260620 03:10:33.288313 19788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5e6dcb476a014f398abfacfc165b5507"
format_stamp: "Formatted at 2026-06-20 03:10:33 on dist-test-slave-xv68"
server_key: "a952344c3ca8daf4773a376acba0da2f"
server_key_iv: "69da616ecfba28b01ec0a5e0b517eaac"
server_key_version: "encryptionkey@0"
I20260620 03:10:33.293232 19788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:10:33.296545 19803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:33.297926 19788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:10:33.298089 19788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "5e6dcb476a014f398abfacfc165b5507"
format_stamp: "Formatted at 2026-06-20 03:10:33 on dist-test-slave-xv68"
server_key: "a952344c3ca8daf4773a376acba0da2f"
server_key_iv: "69da616ecfba28b01ec0a5e0b517eaac"
server_key_version: "encryptionkey@0"
I20260620 03:10:33.298280 19788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:33.318518 19788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:33.322590 19788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:33.322898 19788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:33.323899 19788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:33.325178 19788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:33.325295 19788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:33.325438 19788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:33.325508 19788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:33.359513 19788 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34901
I20260620 03:10:33.359597 19915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34901 every 8 connection(s)
I20260620 03:10:33.360826 19788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:10:33.368355 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 19788
I20260620 03:10:33.368522 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:10:33.368968 388 external_mini_cluster.cc:1428] Setting key 83781e661682f0de5d101d40e18af005
Jun 20 03:10:33 dist-test-slave-xv68 krb5kdc[19681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:10:33.384065 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:33.363357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39375 (local address 127.0.97.62:44995)
0620 03:10:33.363850 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:33.363855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:33.364602 (+ 747us) server_negotiation.cc:407] Connection header received
0620 03:10:33.366442 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:33.366449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:33.366541 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:33.366660 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:33.368328 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:33.369204 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:33.371761 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:33.372558 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:33.372730 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:33.377862 (+ 5132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:33.377867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:33.377871 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:33.377903 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:33.380970 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:33.381719 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:33.381724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:33.381728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:33.381858 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:33.382297 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:33.382302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:33.382306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:33.382575 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:33.382726 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:33.383453 (+ 727us) server_negotiation.cc:299] Negotiation successful
0620 03:10:33.383682 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":260,"threads_started":1}
I20260620 03:10:33.385062 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:33.363745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44995 (local address 127.0.97.1:39375)
0620 03:10:33.364448 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:33.364487 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:33.366171 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:33.366892 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:33.366899 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:33.367480 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:10:33.368122 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:33.368150 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:33.369353 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:33.369358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:33.371587 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:33.371605 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:33.372863 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:33.372872 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:33.373308 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:33.374150 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:10:33.374188 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:10:33.377530 (+ 3342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:10:33.381122 (+ 3592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:10:33.381131 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:10:33.381148 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:10:33.381587 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:10:33.381986 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:10:33.381990 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:10:33.381994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:10:33.382178 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:10:33.382701 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:10:33.382709 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:10:33.383056 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 03:10:33.383339 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 03:10:33.383877 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":265,"threads_started":1}
I20260620 03:10:33.386715 19916 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44995
I20260620 03:10:33.387048 19916 heartbeater.cc:464] Registering TS with master...
I20260620 03:10:33.387828 19916 heartbeater.cc:511] Master 127.0.97.62:44995 requested a full tablet report, sending...
I20260620 03:10:33.390162 19726 ts_manager.cc:194] Registered new tserver with Master: 5e6dcb476a014f398abfacfc165b5507 (127.0.97.1:34901)
I20260620 03:10:33.392422 19726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39375
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:10:33.430848 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44995
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:33.745728 19922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:33.745985 19922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:33.746079 19922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:33.753080 19922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:33.753182 19922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:33.753307 19922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:10:33.762578 19922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:33.763837 19922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:33.765062 19922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:33.779961 19928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:33.779983 19927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:33.780870 19930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:33.781419 19922 server_base.cc:1034] running on GCE node
I20260620 03:10:33.781961 19922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:33.782829 19922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:33.784026 19922 hybrid_clock.cc:648] HybridClock initialized: now 1781925033783987 us; error 34 us; skew 500 ppm
I20260620 03:10:33.784322 19922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:33 dist-test-slave-xv68 krb5kdc[19681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:33.787352 19922 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:10:33.788672 19922 webserver.cc:460] Webserver started at http://127.0.97.2:41715/ using document root <none> and password file <none>
I20260620 03:10:33.789340 19922 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:33.789479 19922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:33.789772 19922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:33.792694 19922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "3a9eca69648747d8b80b8baa7d9cef5f"
format_stamp: "Formatted at 2026-06-20 03:10:33 on dist-test-slave-xv68"
server_key: "c7f5d6508869eb144b28ceab25c30ada"
server_key_iv: "10cabdc4ff22050bd1ec70d6c6ac14c6"
server_key_version: "encryptionkey@0"
I20260620 03:10:33.793444 19922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3a9eca69648747d8b80b8baa7d9cef5f"
format_stamp: "Formatted at 2026-06-20 03:10:33 on dist-test-slave-xv68"
server_key: "c7f5d6508869eb144b28ceab25c30ada"
server_key_iv: "10cabdc4ff22050bd1ec70d6c6ac14c6"
server_key_version: "encryptionkey@0"
I20260620 03:10:33.798488 19922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 03:10:33.801932 19937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:33.803275 19922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:10:33.803503 19922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "3a9eca69648747d8b80b8baa7d9cef5f"
format_stamp: "Formatted at 2026-06-20 03:10:33 on dist-test-slave-xv68"
server_key: "c7f5d6508869eb144b28ceab25c30ada"
server_key_iv: "10cabdc4ff22050bd1ec70d6c6ac14c6"
server_key_version: "encryptionkey@0"
I20260620 03:10:33.803747 19922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:33.820611 19922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:33.824601 19922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:33.824838 19922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:33.825697 19922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:33.826920 19922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:33.826990 19922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:33.827070 19922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:33.827113 19922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:33.860003 19922 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40151
I20260620 03:10:33.860092 20049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40151 every 8 connection(s)
I20260620 03:10:33.861235 19922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:10:33.863637 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 19922
I20260620 03:10:33.863847 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:10:33.864372 388 external_mini_cluster.cc:1428] Setting key eddffc7aa243c13e6102e4810fe920f0
Jun 20 03:10:33 dist-test-slave-xv68 krb5kdc[19681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:10:33.881103 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:33.863812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41613 (local address 127.0.97.62:44995)
0620 03:10:33.864015 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:33.864020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:33.865444 (+ 1424us) server_negotiation.cc:407] Connection header received
0620 03:10:33.867265 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:33.867271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:33.867393 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:33.867499 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:33.868901 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:33.869614 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:33.871093 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:33.871571 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:33.871707 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:33.875512 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:33.875517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:33.875521 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:33.875551 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:33.878011 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:33.878903 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:33.878907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:33.878909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:33.879025 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:33.879504 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:33.879508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:33.879510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:33.879750 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:33.879858 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:33.880685 (+ 827us) server_negotiation.cc:299] Negotiation successful
0620 03:10:33.880845 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 03:10:33.882534 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:33.864445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44995 (local address 127.0.97.2:41613)
0620 03:10:33.865288 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:33.865328 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:33.867021 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:33.867716 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:33.867724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:33.868215 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:10:33.868734 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:33.868751 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:33.869768 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:33.869773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:33.870952 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:33.870963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:33.871788 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:33.871795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:33.872012 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:33.872670 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:10:33.872691 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:10:33.875277 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:10:33.878145 (+ 2868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:10:33.878156 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:10:33.878173 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:10:33.878779 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:10:33.879131 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:10:33.879139 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:10:33.879146 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:10:33.879404 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:10:33.879873 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:10:33.879884 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:10:33.880307 (+ 423us) client_negotiation.cc:769] Sending connection context
0620 03:10:33.880612 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 03:10:33.881155 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":296,"threads_started":1}
I20260620 03:10:33.883944 20050 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44995
I20260620 03:10:33.884276 20050 heartbeater.cc:464] Registering TS with master...
I20260620 03:10:33.884975 20050 heartbeater.cc:511] Master 127.0.97.62:44995 requested a full tablet report, sending...
I20260620 03:10:33.886613 19726 ts_manager.cc:194] Registered new tserver with Master: 3a9eca69648747d8b80b8baa7d9cef5f (127.0.97.2:40151)
I20260620 03:10:33.887859 19726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41613
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:10:33.920513 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44995
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:34.244443 20055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:34.244814 20055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:34.244974 20055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:34.252449 20055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:34.252594 20055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:34.252770 20055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:10:34.262368 20055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:34.263697 20055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:34.264950 20055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:34.277948 20061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:34.278084 20060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:34.278023 20063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:34.278852 20055 server_base.cc:1034] running on GCE node
I20260620 03:10:34.279511 20055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:34.280421 20055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:34.281651 20055 hybrid_clock.cc:648] HybridClock initialized: now 1781925034281610 us; error 49 us; skew 500 ppm
I20260620 03:10:34.281975 20055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:34 dist-test-slave-xv68 krb5kdc[19681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:34.285152 20055 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:10:34.286551 20055 webserver.cc:460] Webserver started at http://127.0.97.3:42811/ using document root <none> and password file <none>
I20260620 03:10:34.287168 20055 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:34.287272 20055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:34.287598 20055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:34.290329 20055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "307781a1cda44f92ab8ab0e6995458da"
format_stamp: "Formatted at 2026-06-20 03:10:34 on dist-test-slave-xv68"
server_key: "a474732f42a17139c45f925c5ee7fe69"
server_key_iv: "ae6cbba7dcfd754872976189e879587a"
server_key_version: "encryptionkey@0"
I20260620 03:10:34.291038 20055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "307781a1cda44f92ab8ab0e6995458da"
format_stamp: "Formatted at 2026-06-20 03:10:34 on dist-test-slave-xv68"
server_key: "a474732f42a17139c45f925c5ee7fe69"
server_key_iv: "ae6cbba7dcfd754872976189e879587a"
server_key_version: "encryptionkey@0"
I20260620 03:10:34.295868 20055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 03:10:34.299084 20070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:34.300379 20055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:10:34.300537 20055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "307781a1cda44f92ab8ab0e6995458da"
format_stamp: "Formatted at 2026-06-20 03:10:34 on dist-test-slave-xv68"
server_key: "a474732f42a17139c45f925c5ee7fe69"
server_key_iv: "ae6cbba7dcfd754872976189e879587a"
server_key_version: "encryptionkey@0"
I20260620 03:10:34.300729 20055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:34.318727 20055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:34.322574 20055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:34.322854 20055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:34.323889 20055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:34.325165 20055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:34.325235 20055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:34.325431 20055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:34.325482 20055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:34.361044 20055 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42061
I20260620 03:10:34.361079 20182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42061 every 8 connection(s)
I20260620 03:10:34.362339 20055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:10:34.364356 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 20055
I20260620 03:10:34.364542 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:10:34.365021 388 external_mini_cluster.cc:1428] Setting key 8e5e5905688b5b13ee75b87674cdd443
Jun 20 03:10:34 dist-test-slave-xv68 krb5kdc[19681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:10:34.382580 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.365257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57125 (local address 127.0.97.62:44995)
0620 03:10:34.365488 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:34.365494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:34.366639 (+ 1145us) server_negotiation.cc:407] Connection header received
0620 03:10:34.368466 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:34.368470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:34.368534 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:34.368622 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:34.370022 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.370772 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.372404 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.372834 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.372939 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:34.376413 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:34.376418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:34.376421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:34.376454 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:34.379614 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:34.380470 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:34.380474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:34.380476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:34.380595 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:34.381048 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:34.381057 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:34.381061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:34.381375 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:34.381492 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:34.382176 (+ 684us) server_negotiation.cc:299] Negotiation successful
0620 03:10:34.382336 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 03:10:34.383476 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.365690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44995 (local address 127.0.97.3:57125)
0620 03:10:34.366492 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:34.366544 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:34.368222 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:34.368840 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:34.368848 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:34.369337 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:10:34.369864 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:34.369880 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.370915 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.370919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:34.372257 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:34.372269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.373031 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.373036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:34.373243 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:34.373970 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:10:34.373997 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:10:34.376195 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:10:34.379745 (+ 3550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:10:34.379755 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:10:34.379770 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:10:34.380341 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:10:34.380708 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:10:34.380712 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:10:34.380715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:10:34.380901 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:10:34.381492 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:10:34.381499 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:10:34.381798 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 03:10:34.382033 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 03:10:34.382385 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":345,"threads_started":1}
I20260620 03:10:34.384692 20183 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44995
I20260620 03:10:34.385022 20183 heartbeater.cc:464] Registering TS with master...
I20260620 03:10:34.385754 20183 heartbeater.cc:511] Master 127.0.97.62:44995 requested a full tablet report, sending...
I20260620 03:10:34.387212 19726 ts_manager.cc:194] Registered new tserver with Master: 307781a1cda44f92ab8ab0e6995458da (127.0.97.3:42061)
I20260620 03:10:34.388077 19726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57125
I20260620 03:10:34.392439 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:10:34.396399 19916 heartbeater.cc:503] Master 127.0.97.62:44995 was elected leader, sending a full tablet report...
I20260620 03:10:34.409523 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.396698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.0.97.62:44995)
0620 03:10:34.396920 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:34.396926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:34.397310 (+ 384us) server_negotiation.cc:407] Connection header received
0620 03:10:34.397452 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:34.397457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:34.397524 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:34.397625 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:34.399086 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.399807 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.401034 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.401429 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.401588 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:34.404171 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:34.404176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:34.404180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:34.404208 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:34.407144 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:34.407790 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:34.407794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:34.407796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:34.407916 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:34.408289 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:34.408293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:34.408295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:34.408628 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:34.408808 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:34.409138 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 03:10:34.409292 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 03:10:34.413501 19726 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:10:34.420317 19726 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:10:34.442327 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.432969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42061 (local address 127.0.0.1:57066)
0620 03:10:34.433464 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:34.433485 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:34.433584 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:34.434859 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:34.434866 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:34.434902 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:34.435465 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:34.435476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.437580 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.437584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:34.439605 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:34.439617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.441431 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.441441 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:34.441626 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:34.441673 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 03:10:34.441782 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 03:10:34.441931 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":245,"threads_started":1}
I20260620 03:10:34.442623 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.431929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60444 (local address 127.0.97.2:40151)
0620 03:10:34.432647 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:34.432653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:34.432679 (+ 26us) server_negotiation.cc:407] Connection header received
0620 03:10:34.433005 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:34.433014 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:34.433268 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:34.433460 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:34.435645 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.437305 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.439988 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.441384 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.441505 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:34.441704 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:34.442136 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 03:10:34.442312 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":446,"threads_started":1}
I20260620 03:10:34.443171 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.433551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34901 (local address 127.0.0.1:35540)
0620 03:10:34.434421 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:34.434441 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:34.434557 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:34.436094 (+ 1537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:34.436100 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:34.436129 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:34.436513 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:34.436523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.438695 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.438700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:34.440511 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:34.440521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.442468 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.442475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:34.442607 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:34.442628 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 03:10:34.442734 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 03:10:34.442862 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":608,"threads_started":1}
I20260620 03:10:34.443464 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.434152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35540 (local address 127.0.97.1:34901)
0620 03:10:34.435663 (+ 1511us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:34.435670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:34.435695 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:10:34.435779 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:34.435784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:34.435925 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:34.436120 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:34.436656 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.438547 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.440659 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.442134 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.442243 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:34.442468 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:34.442958 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 03:10:34.443110 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1349,"thread_start_us":1175,"threads_started":1}
I20260620 03:10:34.443648 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.431816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40151 (local address 127.0.0.1:60444)
0620 03:10:34.432555 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:34.432600 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:34.432752 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:34.433742 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:34.433750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:34.433781 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:34.435046 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:34.435062 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.437479 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.437487 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:34.439825 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:34.439839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.441757 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.441763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:34.441940 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:34.441974 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 03:10:34.442074 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 03:10:34.442200 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":1,"thread_start_us":376,"threads_started":1}
I20260620 03:10:34.444061 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.433282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57066 (local address 127.0.97.3:42061)
0620 03:10:34.434170 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:34.434177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:34.434226 (+ 49us) server_negotiation.cc:407] Connection header received
0620 03:10:34.434339 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:34.434349 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:34.434605 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:34.435668 (+ 1063us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:34.435934 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.437433 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.439758 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.441179 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.443038 (+ 1859us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:34.443193 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:34.443458 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 03:10:34.443656 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":522,"threads_started":1}
I20260620 03:10:34.446902 19985 tablet_service.cc:1467] Processing CreateTablet for tablet a99047877ca04ba1b76a989aa673c7c6 (DEFAULT_TABLE table=test-table [id=62d98c19d4644d9ebbb11a8816a04d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:10:34.447139 19851 tablet_service.cc:1467] Processing CreateTablet for tablet a99047877ca04ba1b76a989aa673c7c6 (DEFAULT_TABLE table=test-table [id=62d98c19d4644d9ebbb11a8816a04d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:10:34.447139 20118 tablet_service.cc:1467] Processing CreateTablet for tablet a99047877ca04ba1b76a989aa673c7c6 (DEFAULT_TABLE table=test-table [id=62d98c19d4644d9ebbb11a8816a04d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:10:34.448174 19985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a99047877ca04ba1b76a989aa673c7c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:34.448546 19851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a99047877ca04ba1b76a989aa673c7c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:34.448738 20118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a99047877ca04ba1b76a989aa673c7c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:34.459093 20201 tablet_bootstrap.cc:492] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f: Bootstrap starting.
I20260620 03:10:34.462615 20202 tablet_bootstrap.cc:492] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507: Bootstrap starting.
I20260620 03:10:34.463624 20203 tablet_bootstrap.cc:492] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da: Bootstrap starting.
I20260620 03:10:34.464114 20201 tablet_bootstrap.cc:654] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:34.465344 20201 log.cc:824] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:34.466053 20202 tablet_bootstrap.cc:654] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:34.466459 20203 tablet_bootstrap.cc:654] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:34.467159 20202 log.cc:824] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:34.467434 20203 log.cc:824] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:34.468466 20201 tablet_bootstrap.cc:492] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f: No bootstrap required, opened a new log
I20260620 03:10:34.468734 20201 ts_tablet_manager.cc:1397] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20260620 03:10:34.470081 20202 tablet_bootstrap.cc:492] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507: No bootstrap required, opened a new log
I20260620 03:10:34.470180 20203 tablet_bootstrap.cc:492] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da: No bootstrap required, opened a new log
I20260620 03:10:34.470368 20202 ts_tablet_manager.cc:1397] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 03:10:34.470379 20203 ts_tablet_manager.cc:1397] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 03:10:34.476651 20201 raft_consensus.cc:348] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9eca69648747d8b80b8baa7d9cef5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40151 } } peers { permanent_uuid: "307781a1cda44f92ab8ab0e6995458da" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42061 } } peers { permanent_uuid: "5e6dcb476a014f398abfacfc165b5507" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34901 } }
I20260620 03:10:34.477013 20201 raft_consensus.cc:374] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:34.477118 20201 raft_consensus.cc:729] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9eca69648747d8b80b8baa7d9cef5f, State: Initialized, Role: FOLLOWER
I20260620 03:10:34.477715 20201 consensus_queue.cc:260] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9eca69648747d8b80b8baa7d9cef5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40151 } } peers { permanent_uuid: "307781a1cda44f92ab8ab0e6995458da" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42061 } } peers { permanent_uuid: "5e6dcb476a014f398abfacfc165b5507" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34901 } }
I20260620 03:10:34.478760 20050 heartbeater.cc:503] Master 127.0.97.62:44995 was elected leader, sending a full tablet report...
I20260620 03:10:34.479422 20201 ts_tablet_manager.cc:1428] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f: Time spent starting tablet: real 0.011s user 0.003s sys 0.009s
I20260620 03:10:34.480875 20203 raft_consensus.cc:348] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9eca69648747d8b80b8baa7d9cef5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40151 } } peers { permanent_uuid: "307781a1cda44f92ab8ab0e6995458da" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42061 } } peers { permanent_uuid: "5e6dcb476a014f398abfacfc165b5507" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34901 } }
I20260620 03:10:34.481334 20203 raft_consensus.cc:374] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:34.481477 20203 raft_consensus.cc:729] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307781a1cda44f92ab8ab0e6995458da, State: Initialized, Role: FOLLOWER
I20260620 03:10:34.481665 20202 raft_consensus.cc:348] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9eca69648747d8b80b8baa7d9cef5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40151 } } peers { permanent_uuid: "307781a1cda44f92ab8ab0e6995458da" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42061 } } peers { permanent_uuid: "5e6dcb476a014f398abfacfc165b5507" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34901 } }
I20260620 03:10:34.482117 20202 raft_consensus.cc:374] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:34.482072 20203 consensus_queue.cc:260] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9eca69648747d8b80b8baa7d9cef5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40151 } } peers { permanent_uuid: "307781a1cda44f92ab8ab0e6995458da" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42061 } } peers { permanent_uuid: "5e6dcb476a014f398abfacfc165b5507" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34901 } }
I20260620 03:10:34.482249 20202 raft_consensus.cc:729] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e6dcb476a014f398abfacfc165b5507, State: Initialized, Role: FOLLOWER
I20260620 03:10:34.483495 20183 heartbeater.cc:503] Master 127.0.97.62:44995 was elected leader, sending a full tablet report...
I20260620 03:10:34.483472 20202 consensus_queue.cc:260] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9eca69648747d8b80b8baa7d9cef5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40151 } } peers { permanent_uuid: "307781a1cda44f92ab8ab0e6995458da" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42061 } } peers { permanent_uuid: "5e6dcb476a014f398abfacfc165b5507" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34901 } }
I20260620 03:10:34.484129 20203 ts_tablet_manager.cc:1428] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260620 03:10:34.485638 20202 ts_tablet_manager.cc:1428] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20260620 03:10:34.501415 20208 raft_consensus.cc:482] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:10:34.501710 20208 raft_consensus.cc:504] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9eca69648747d8b80b8baa7d9cef5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40151 } } peers { permanent_uuid: "307781a1cda44f92ab8ab0e6995458da" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42061 } } peers { permanent_uuid: "5e6dcb476a014f398abfacfc165b5507" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34901 } }
I20260620 03:10:34.503283 20208 leader_election.cc:288] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a9eca69648747d8b80b8baa7d9cef5f (127.0.97.2:40151), 5e6dcb476a014f398abfacfc165b5507 (127.0.97.1:34901)
I20260620 03:10:34.509358 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.503568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40151 (local address 127.0.97.3:46477)
0620 03:10:34.503760 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:34.503789 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:34.503905 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:34.504324 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:34.504331 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:34.504364 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:34.504563 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:34.504573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.505839 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.505844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:34.507432 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:34.507443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.508627 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.508633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:34.508813 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:34.508835 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 03:10:34.508947 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 03:10:34.509073 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 03:10:34.509370 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.503781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46477 (local address 127.0.97.2:40151)
0620 03:10:34.503979 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:34.503986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:34.504007 (+ 21us) server_negotiation.cc:407] Connection header received
0620 03:10:34.504081 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:34.504105 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:34.504187 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:34.504285 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:34.504703 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.505704 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.507568 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.508469 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.508607 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:34.508697 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:34.508973 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 03:10:34.509092 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 03:10:34.510011 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.504083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52213 (local address 127.0.97.1:34901)
0620 03:10:34.504317 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:34.504324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:34.504621 (+ 297us) server_negotiation.cc:407] Connection header received
0620 03:10:34.504817 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:34.504823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:34.504898 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:34.504994 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:34.505563 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.506370 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.508238 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.509081 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.509228 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:34.509340 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:34.509654 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 03:10:34.509774 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 03:10:34.510133 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.503912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34901 (local address 127.0.97.3:52213)
0620 03:10:34.504527 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:34.504542 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:34.504678 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:34.505021 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:34.505026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:34.505051 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:34.505408 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:34.505418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.506511 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.506516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:34.508096 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:34.508107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.509291 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.509297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:34.509497 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:34.509531 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 03:10:34.509616 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 03:10:34.509731 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":2,"thread_start_us":410,"threads_started":1}
I20260620 03:10:34.510694 20005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99047877ca04ba1b76a989aa673c7c6" candidate_uuid: "307781a1cda44f92ab8ab0e6995458da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9eca69648747d8b80b8baa7d9cef5f" is_pre_election: true
I20260620 03:10:34.511087 19871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99047877ca04ba1b76a989aa673c7c6" candidate_uuid: "307781a1cda44f92ab8ab0e6995458da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e6dcb476a014f398abfacfc165b5507" is_pre_election: true
I20260620 03:10:34.511216 20005 raft_consensus.cc:2454] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 307781a1cda44f92ab8ab0e6995458da in term 0.
I20260620 03:10:34.511593 19871 raft_consensus.cc:2454] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 307781a1cda44f92ab8ab0e6995458da in term 0.
I20260620 03:10:34.511904 20073 leader_election.cc:302] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 307781a1cda44f92ab8ab0e6995458da, 3a9eca69648747d8b80b8baa7d9cef5f; no voters:
I20260620 03:10:34.512260 20208 raft_consensus.cc:2781] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:10:34.512418 20208 raft_consensus.cc:482] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:10:34.512529 20208 raft_consensus.cc:3037] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:34.515067 20208 raft_consensus.cc:504] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9eca69648747d8b80b8baa7d9cef5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40151 } } peers { permanent_uuid: "307781a1cda44f92ab8ab0e6995458da" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42061 } } peers { permanent_uuid: "5e6dcb476a014f398abfacfc165b5507" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34901 } }
I20260620 03:10:34.515722 20208 leader_election.cc:288] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [CANDIDATE]: Term 1 election: Requested vote from peers 3a9eca69648747d8b80b8baa7d9cef5f (127.0.97.2:40151), 5e6dcb476a014f398abfacfc165b5507 (127.0.97.1:34901)
I20260620 03:10:34.516044 20005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99047877ca04ba1b76a989aa673c7c6" candidate_uuid: "307781a1cda44f92ab8ab0e6995458da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9eca69648747d8b80b8baa7d9cef5f"
I20260620 03:10:34.516129 19871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99047877ca04ba1b76a989aa673c7c6" candidate_uuid: "307781a1cda44f92ab8ab0e6995458da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e6dcb476a014f398abfacfc165b5507"
I20260620 03:10:34.516228 20005 raft_consensus.cc:3037] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:34.516358 19871 raft_consensus.cc:3037] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:34.518492 20005 raft_consensus.cc:2454] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 307781a1cda44f92ab8ab0e6995458da in term 1.
I20260620 03:10:34.518515 19871 raft_consensus.cc:2454] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 307781a1cda44f92ab8ab0e6995458da in term 1.
I20260620 03:10:34.518832 20073 leader_election.cc:302] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 307781a1cda44f92ab8ab0e6995458da, 5e6dcb476a014f398abfacfc165b5507; no voters:
I20260620 03:10:34.519125 20208 raft_consensus.cc:2781] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:34.519591 20208 raft_consensus.cc:686] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [term 1 LEADER]: Becoming Leader. State: Replica: 307781a1cda44f92ab8ab0e6995458da, State: Running, Role: LEADER
I20260620 03:10:34.520296 20208 consensus_queue.cc:237] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9eca69648747d8b80b8baa7d9cef5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40151 } } peers { permanent_uuid: "307781a1cda44f92ab8ab0e6995458da" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42061 } } peers { permanent_uuid: "5e6dcb476a014f398abfacfc165b5507" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34901 } }
I20260620 03:10:34.524667 19726 catalog_manager.cc:5495] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da reported cstate change: term changed from 0 to 1, leader changed from <none> to 307781a1cda44f92ab8ab0e6995458da (127.0.97.3). New cstate: current_term: 1 leader_uuid: "307781a1cda44f92ab8ab0e6995458da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9eca69648747d8b80b8baa7d9cef5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "307781a1cda44f92ab8ab0e6995458da" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e6dcb476a014f398abfacfc165b5507" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34901 } health_report { overall_health: UNKNOWN } } }
I20260620 03:10:34.544461 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.539746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57080 (local address 127.0.97.3:42061)
0620 03:10:34.539933 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:34.539937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:34.539955 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:10:34.540026 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:34.540029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:34.540100 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:34.540214 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:10:34.540519 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.541220 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.542556 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.542938 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.543025 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:34.543467 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:10:34.543935 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:10:34.544038 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:34.544124 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 03:10:34.544183 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 03:10:34.552222 20005 raft_consensus.cc:1261] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f [term 1 FOLLOWER]: Refusing update from remote peer 307781a1cda44f92ab8ab0e6995458da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:10:34.552230 19871 raft_consensus.cc:1261] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507 [term 1 FOLLOWER]: Refusing update from remote peer 307781a1cda44f92ab8ab0e6995458da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:10:34.553122 20208 consensus_queue.cc:1035] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9eca69648747d8b80b8baa7d9cef5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:10:34.553476 20211 consensus_queue.cc:1035] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e6dcb476a014f398abfacfc165b5507" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:10:34.563089 20216 mvcc.cc:205] Tried to move back new op lower bound from 7298764941516140544 to 7298764941398937600. Current Snapshot: MvccSnapshot[applied={T|T < 7298764941516140544}]
I20260620 03:10:34.566381 20214 mvcc.cc:205] Tried to move back new op lower bound from 7298764941516140544 to 7298764941398937600. Current Snapshot: MvccSnapshot[applied={T|T < 7298764941516140544}]
I20260620 03:10:34.579023 19726 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:10:34.579317 19726 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:10:34.582845 19726 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a69eb360175640a6bd74282652cdeb2e: Sending DeleteTablet for 3 replicas of tablet a99047877ca04ba1b76a989aa673c7c6
I20260620 03:10:34.584000 19985 tablet_service.cc:1514] Processing DeleteTablet for tablet a99047877ca04ba1b76a989aa673c7c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:10:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60444
I20260620 03:10:34.584048 20118 tablet_service.cc:1514] Processing DeleteTablet for tablet a99047877ca04ba1b76a989aa673c7c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:10:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57066
I20260620 03:10:34.584173 19851 tablet_service.cc:1514] Processing DeleteTablet for tablet a99047877ca04ba1b76a989aa673c7c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:10:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35540
I20260620 03:10:34.586091 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 19788
I20260620 03:10:34.596412 20225 tablet_replica.cc:330] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f: stopping tablet replica
I20260620 03:10:34.597025 20224 tablet_replica.cc:330] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da: stopping tablet replica
I20260620 03:10:34.597774 20224 raft_consensus.cc:2229] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:10:34.598374 20225 raft_consensus.cc:2229] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:10:34.599089 20225 raft_consensus.cc:2258] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:10:34.599907 20224 raft_consensus.cc:2258] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:10:34.603111 20224 ts_tablet_manager.cc:1905] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:10:34.603186 20226 tablet_replica.cc:330] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507: stopping tablet replica
I20260620 03:10:34.603572 20226 raft_consensus.cc:2229] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:10:34.604035 20225 ts_tablet_manager.cc:1905] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:10:34.605376 20226 raft_consensus.cc:2258] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 03:10:34 dist-test-slave-xv68 krb5kdc[19681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:10:34.608819 20226 ts_tablet_manager.cc:1905] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:10:34.611155 20224 ts_tablet_manager.cc:1918] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:10:34.611425 20224 log.cc:1196] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/a99047877ca04ba1b76a989aa673c7c6
I20260620 03:10:34.612135 20224 ts_tablet_manager.cc:1939] T a99047877ca04ba1b76a989aa673c7c6 P 307781a1cda44f92ab8ab0e6995458da: Deleting consensus metadata
I20260620 03:10:34.613984 19712 catalog_manager.cc:4841] TS 307781a1cda44f92ab8ab0e6995458da (127.0.97.3:42061): tablet a99047877ca04ba1b76a989aa673c7c6 (table test-table [id=62d98c19d4644d9ebbb11a8816a04d7d]) successfully deleted
I20260620 03:10:34.615270 20225 ts_tablet_manager.cc:1918] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:10:34.615541 20225 log.cc:1196] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/a99047877ca04ba1b76a989aa673c7c6
I20260620 03:10:34.616137 20225 ts_tablet_manager.cc:1939] T a99047877ca04ba1b76a989aa673c7c6 P 3a9eca69648747d8b80b8baa7d9cef5f: Deleting consensus metadata
I20260620 03:10:34.617281 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.587825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.0.97.1:34901)
0620 03:10:34.588045 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:34.588054 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:34.597206 (+ 9152us) server_negotiation.cc:407] Connection header received
0620 03:10:34.597481 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:34.597488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:34.597596 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:34.597670 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:34.599407 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.600647 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.602490 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.603195 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.603335 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:34.609456 (+ 6121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:34.609469 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:34.609484 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:34.609530 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:34.613518 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:34.614455 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:34.614467 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:34.614470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:34.614620 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:34.615096 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:34.615101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:34.615104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:34.615644 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:34.616003 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:34.616633 (+ 630us) server_negotiation.cc:299] Negotiation successful
0620 03:10:34.616862 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:10:34.618189 19712 catalog_manager.cc:4841] TS 3a9eca69648747d8b80b8baa7d9cef5f (127.0.97.2:40151): tablet a99047877ca04ba1b76a989aa673c7c6 (table test-table [id=62d98c19d4644d9ebbb11a8816a04d7d]) successfully deleted
I20260620 03:10:34.619158 19911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:34.707713 20226 ts_tablet_manager.cc:1918] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:10:34.708221 20226 log.cc:1196] T a99047877ca04ba1b76a989aa673c7c6 P 5e6dcb476a014f398abfacfc165b5507: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924720365844-388-0/minicluster-data/ts-0/wal/wals/a99047877ca04ba1b76a989aa673c7c6
I20260620 03:10:34.708380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 19788
W20260620 03:10:34.732029 19712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:34901 (error 108)
I20260620 03:10:34.732816 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 19922
I20260620 03:10:34.734220 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.733726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34901 (local address 127.0.0.1:35562)
0620 03:10:34.733939 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:34.734051 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 03:10:34.734448 19712 catalog_manager.cc:4584] TS 5e6dcb476a014f398abfacfc165b5507 (127.0.97.1:34901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a99047877ca04ba1b76a989aa673c7c6: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34901: connect: Connection refused (error 111)
Jun 20 03:10:34 dist-test-slave-xv68 krb5kdc[19681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:10:34.748365 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.733599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.0.97.2:40151)
0620 03:10:34.734003 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:34.734009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:34.734030 (+ 21us) server_negotiation.cc:407] Connection header received
0620 03:10:34.734135 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:34.734140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:34.734251 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:34.734349 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:34.735935 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.736662 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.737908 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.738300 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.738439 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:34.742893 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:34.742905 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:34.742918 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:34.742947 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:34.745710 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:34.746396 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:34.746402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:34.746404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:34.746523 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:34.746879 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:34.746885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:34.746889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:34.747236 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:34.747607 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:34.747838 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 03:10:34.748044 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278}
I20260620 03:10:34.749074 20045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:34.788152 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.787606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34901 (local address 127.0.0.1:35572)
0620 03:10:34.787862 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:34.787973 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260620 03:10:34.789462 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.789056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34901 (local address 127.0.0.1:35578)
0620 03:10:34.789246 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:34.789336 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 03:10:34.838965 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 19922
I20260620 03:10:34.850219 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.849637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34901 (local address 127.0.0.1:35592)
0620 03:10:34.849898 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:34.850007 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 03:10:34.851531 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.851021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34901 (local address 127.0.0.1:35594)
0620 03:10:34.851214 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:34.851357 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 03:10:34.856856 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 20055
Jun 20 03:10:34 dist-test-slave-xv68 krb5kdc[19681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:10:34.875236 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.857744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57094 (local address 127.0.97.3:42061)
0620 03:10:34.857941 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:34.857949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:34.857976 (+ 27us) server_negotiation.cc:407] Connection header received
0620 03:10:34.858043 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:34.858048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:34.858142 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:34.858276 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:34.859660 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.860891 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.862219 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:34.862942 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:34.863083 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:34.867785 (+ 4702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:34.867799 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:34.867819 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:34.867875 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:34.871984 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:34.872734 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:34.872744 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:34.872748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:34.873006 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:34.873443 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:34.873448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:34.873451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:34.873941 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:34.874231 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:34.874558 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 03:10:34.874788 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 03:10:34.876083 20178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:34.932099 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.931618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34901 (local address 127.0.0.1:35596)
0620 03:10:34.931832 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:34.931936 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 03:10:34.933193 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:34.932762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34901 (local address 127.0.0.1:35612)
0620 03:10:34.933019 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:34.933087 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260620 03:10:35.000455 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 20055
I20260620 03:10:35.020040 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 19694
I20260620 03:10:35.021261 19756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:35.130759 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 19694
I20260620 03:10:35.130806 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:35.130236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34901 (local address 127.0.0.1:35614)
0620 03:10:35.130454 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:35.130574 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
2026-06-20T03:10:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4135 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T03:10:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:10:35Z Disabled control of system clock
I20260620 03:10:35.206878 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36191
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 03:10:35.535609 20236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:35.535889 20236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:35.535970 20236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:35.543031 20236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:10:35.543136 20236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:35.543175 20236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:35.543226 20236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:10:35.543277 20236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:10:35.552412 20236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36191
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:35.553617 20236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:35.555326 20236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:35.567394 20241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:35.567420 20242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:35.567374 20244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:35.568302 20236 server_base.cc:1034] running on GCE node
I20260620 03:10:35.569157 20236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:35.570607 20236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:35.571852 20236 hybrid_clock.cc:648] HybridClock initialized: now 1781925035571816 us; error 69 us; skew 500 ppm
I20260620 03:10:35.572228 20236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:10:35.574467 20236 webserver.cc:460] Webserver started at http://127.0.97.62:37783/ using document root <none> and password file <none>
I20260620 03:10:35.575110 20236 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:35.575194 20236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:35.575548 20236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:35.578527 20236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "ca654b78377049ffb8bea8ddfe1c9329"
format_stamp: "Formatted at 2026-06-20 03:10:35 on dist-test-slave-xv68"
server_key: "325050435e0b9037d7fb5084fd770168"
server_key_iv: "3a1aa6a9babd412d39d69481af38a19b"
server_key_version: "encryptionkey@0"
I20260620 03:10:35.579402 20236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "ca654b78377049ffb8bea8ddfe1c9329"
format_stamp: "Formatted at 2026-06-20 03:10:35 on dist-test-slave-xv68"
server_key: "325050435e0b9037d7fb5084fd770168"
server_key_iv: "3a1aa6a9babd412d39d69481af38a19b"
server_key_version: "encryptionkey@0"
I20260620 03:10:35.584517 20236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 03:10:35.587836 20250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:35.589412 20236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:10:35.589607 20236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "ca654b78377049ffb8bea8ddfe1c9329"
format_stamp: "Formatted at 2026-06-20 03:10:35 on dist-test-slave-xv68"
server_key: "325050435e0b9037d7fb5084fd770168"
server_key_iv: "3a1aa6a9babd412d39d69481af38a19b"
server_key_version: "encryptionkey@0"
I20260620 03:10:35.589823 20236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:35.609699 20236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:35.610518 20236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:35.610813 20236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:35.629508 20236 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36191
I20260620 03:10:35.629529 20301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36191 every 8 connection(s)
I20260620 03:10:35.630895 20236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:10:35.634500 20302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:35.640311 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 20236
I20260620 03:10:35.640522 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:10:35.640938 388 external_mini_cluster.cc:1428] Setting key 187a7a697421ba1dfdd17aaed75d2b42
I20260620 03:10:35.644881 20302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329: Bootstrap starting.
I20260620 03:10:35.648433 20302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:35.649771 20302 log.cc:824] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:35.652642 20302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329: No bootstrap required, opened a new log
I20260620 03:10:35.655051 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:35.642847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.0.97.62:36191)
0620 03:10:35.643846 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:35.643857 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:35.643895 (+ 38us) server_negotiation.cc:407] Connection header received
0620 03:10:35.645825 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:35.645848 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:35.646286 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:35.646620 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:35.647247 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:35.648716 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:35.650762 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:35.651508 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:35.651711 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:35.652854 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:35.652862 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:35.652876 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:35.653399 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:35.653479 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:35.653503 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:35.653618 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:35.653849 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 03:10:35.654147 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":393,"threads_started":1}
I20260620 03:10:35.660743 20302 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca654b78377049ffb8bea8ddfe1c9329" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36191 } }
I20260620 03:10:35.661108 20302 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:35.661213 20302 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca654b78377049ffb8bea8ddfe1c9329, State: Initialized, Role: FOLLOWER
I20260620 03:10:35.661792 20302 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca654b78377049ffb8bea8ddfe1c9329" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36191 } }
I20260620 03:10:35.662035 20302 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:10:35.662187 20302 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:10:35.662370 20302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:35.664739 20302 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca654b78377049ffb8bea8ddfe1c9329" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36191 } }
I20260620 03:10:35.665228 20302 leader_election.cc:302] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca654b78377049ffb8bea8ddfe1c9329; no voters:
I20260620 03:10:35.665812 20302 leader_election.cc:288] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:10:35.665984 20307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:35.666834 20307 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329 [term 1 LEADER]: Becoming Leader. State: Replica: ca654b78377049ffb8bea8ddfe1c9329, State: Running, Role: LEADER
I20260620 03:10:35.667402 20307 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca654b78377049ffb8bea8ddfe1c9329" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36191 } }
I20260620 03:10:35.667908 20302 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:10:35.670015 20308 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca654b78377049ffb8bea8ddfe1c9329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca654b78377049ffb8bea8ddfe1c9329" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36191 } } }
I20260620 03:10:35.670126 20309 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca654b78377049ffb8bea8ddfe1c9329. Latest consensus state: current_term: 1 leader_uuid: "ca654b78377049ffb8bea8ddfe1c9329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca654b78377049ffb8bea8ddfe1c9329" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36191 } } }
I20260620 03:10:35.670429 20308 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:35.670511 20309 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:35.671340 20313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:10:35.677680 20313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:10:35.686545 20313 catalog_manager.cc:1347] Generated new cluster ID: d8d06760e9eb4748abf5ac3d1a3e57ee
I20260620 03:10:35.686662 20313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:10:35.694751 20313 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:10:35.696036 20313 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:10:35.716008 20313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329: Generated new TSK 0
I20260620 03:10:35.717120 20313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:10:35.729849 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36191
--builtin_ntp_servers=127.0.97.20:44033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 03:10:36.053210 20326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:36.053483 20326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:36.053615 20326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:36.060784 20326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:36.060886 20326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:36.061053 20326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:10:36.070429 20326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:36.071774 20326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:36.073642 20326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:36.087258 20334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:36.087321 20332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:36.087762 20331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:36.093168 20326 server_base.cc:1034] running on GCE node
I20260620 03:10:36.093848 20326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:36.094877 20326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:36.096105 20326 hybrid_clock.cc:648] HybridClock initialized: now 1781925036096067 us; error 58 us; skew 500 ppm
I20260620 03:10:36.096424 20326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:10:36.098703 20326 webserver.cc:460] Webserver started at http://127.0.97.1:37587/ using document root <none> and password file <none>
I20260620 03:10:36.099462 20326 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:36.099598 20326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:36.099938 20326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:36.102842 20326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-0/data/instance:
uuid: "ae366687b9634d25890b1f0eea4ab447"
format_stamp: "Formatted at 2026-06-20 03:10:36 on dist-test-slave-xv68"
server_key: "d565a63cbfc9a622ea54ee4baff96d30"
server_key_iv: "8d6e823a67c2a0bc477ae11398bd5d4f"
server_key_version: "encryptionkey@0"
I20260620 03:10:36.103694 20326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ae366687b9634d25890b1f0eea4ab447"
format_stamp: "Formatted at 2026-06-20 03:10:36 on dist-test-slave-xv68"
server_key: "d565a63cbfc9a622ea54ee4baff96d30"
server_key_iv: "8d6e823a67c2a0bc477ae11398bd5d4f"
server_key_version: "encryptionkey@0"
I20260620 03:10:36.108506 20326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 03:10:36.111784 20340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:36.113022 20326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:10:36.113189 20326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-0/wal
uuid: "ae366687b9634d25890b1f0eea4ab447"
format_stamp: "Formatted at 2026-06-20 03:10:36 on dist-test-slave-xv68"
server_key: "d565a63cbfc9a622ea54ee4baff96d30"
server_key_iv: "8d6e823a67c2a0bc477ae11398bd5d4f"
server_key_version: "encryptionkey@0"
I20260620 03:10:36.113401 20326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:36.147825 20326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:36.148733 20326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:36.149037 20326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:36.150040 20326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:36.151621 20326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:36.151751 20326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:36.151880 20326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:36.151971 20326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:36.184624 20326 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38499
I20260620 03:10:36.184676 20452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38499 every 8 connection(s)
I20260620 03:10:36.186281 20326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:10:36.194830 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 20326
I20260620 03:10:36.194990 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:10:36.195524 388 external_mini_cluster.cc:1428] Setting key ff4f8c1695e38c08c07ec46185d3471a
I20260620 03:10:36.198444 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36191
--builtin_ntp_servers=127.0.97.20:44033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 03:10:36.199223 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:36.188805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39633 (local address 127.0.97.62:36191)
0620 03:10:36.189201 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:36.189206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:36.190386 (+ 1180us) server_negotiation.cc:407] Connection header received
0620 03:10:36.192198 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:36.192203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:36.192298 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:36.192399 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:36.193499 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:36.194245 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:36.196120 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:36.196662 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:36.196794 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:36.198306 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:36.198312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:36.198317 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:36.198337 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:36.198387 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:36.198405 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:36.198510 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:36.198779 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 03:10:36.198875 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":293,"thread_start_us":198,"threads_started":1}
I20260620 03:10:36.200213 20456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:36.189407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36191 (local address 127.0.97.1:39633)
0620 03:10:36.190240 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:36.190274 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:36.191937 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:36.192592 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:36.192600 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:36.192682 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:10:36.193292 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:36.193311 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:36.194499 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:36.194503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:36.195953 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:36.195970 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:36.196889 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:36.196897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:36.197190 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:36.197987 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:10:36.197990 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:10:36.198030 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:10:36.198104 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:10:36.198509 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:10:36.198517 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:10:36.198522 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 03:10:36.198752 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 03:10:36.199164 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":256,"threads_started":1}
I20260620 03:10:36.201673 20453 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36191
I20260620 03:10:36.201994 20453 heartbeater.cc:464] Registering TS with master...
I20260620 03:10:36.202876 20453 heartbeater.cc:511] Master 127.0.97.62:36191 requested a full tablet report, sending...
I20260620 03:10:36.205150 20267 ts_manager.cc:194] Registered new tserver with Master: ae366687b9634d25890b1f0eea4ab447 (127.0.97.1:38499)
I20260620 03:10:36.207228 20267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:39633
W20260620 03:10:36.525929 20457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:36.526180 20457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:36.526317 20457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:36.533483 20457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:36.533571 20457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:36.533725 20457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:10:36.542363 20457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:36.543730 20457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:36.546535 20457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:36.564675 20463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:36.564813 20465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:36.565438 20462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:36.566999 20457 server_base.cc:1034] running on GCE node
I20260620 03:10:36.567879 20457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:36.569250 20457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:36.570565 20457 hybrid_clock.cc:648] HybridClock initialized: now 1781925036570476 us; error 64 us; skew 500 ppm
I20260620 03:10:36.570911 20457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:10:36.577056 20457 webserver.cc:460] Webserver started at http://127.0.97.2:43731/ using document root <none> and password file <none>
I20260620 03:10:36.577766 20457 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:36.577895 20457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:36.578202 20457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:36.582046 20457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-1/data/instance:
uuid: "c9f6c5b1ed66455c8132abc6c83cb3fb"
format_stamp: "Formatted at 2026-06-20 03:10:36 on dist-test-slave-xv68"
server_key: "c9c082052084f05ebee73f405d629b2b"
server_key_iv: "03dc9a47d51ea05a7e488482d66ea4de"
server_key_version: "encryptionkey@0"
I20260620 03:10:36.583153 20457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c9f6c5b1ed66455c8132abc6c83cb3fb"
format_stamp: "Formatted at 2026-06-20 03:10:36 on dist-test-slave-xv68"
server_key: "c9c082052084f05ebee73f405d629b2b"
server_key_iv: "03dc9a47d51ea05a7e488482d66ea4de"
server_key_version: "encryptionkey@0"
I20260620 03:10:36.590312 20457 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260620 03:10:36.595132 20471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:36.596552 20457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:10:36.596724 20457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-1/wal
uuid: "c9f6c5b1ed66455c8132abc6c83cb3fb"
format_stamp: "Formatted at 2026-06-20 03:10:36 on dist-test-slave-xv68"
server_key: "c9c082052084f05ebee73f405d629b2b"
server_key_iv: "03dc9a47d51ea05a7e488482d66ea4de"
server_key_version: "encryptionkey@0"
I20260620 03:10:36.597000 20457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:36.607904 20457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:36.608608 20457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:36.608862 20457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:36.609736 20457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:36.611083 20457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:36.611164 20457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:36.611327 20457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:36.611436 20457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 03:10:36.647251 20457 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42523
I20260620 03:10:36.647316 20583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42523 every 8 connection(s)
I20260620 03:10:36.648505 20457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:10:36.656775 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 20457
I20260620 03:10:36.656956 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:10:36.657402 388 external_mini_cluster.cc:1428] Setting key e3eaa82f0aaeda7494cd156a7748b101
I20260620 03:10:36.660148 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36191
--builtin_ntp_servers=127.0.97.20:44033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 03:10:36.660630 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:36.650997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58645 (local address 127.0.97.62:36191)
0620 03:10:36.651176 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:36.651180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:36.652467 (+ 1287us) server_negotiation.cc:407] Connection header received
0620 03:10:36.654230 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:36.654235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:36.654329 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:36.654446 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:36.655495 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:36.656221 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:36.657944 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:36.658397 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:36.658552 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:36.659830 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:36.659837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:36.659841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:36.659868 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:36.659904 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:36.659915 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:36.660007 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:36.660234 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 03:10:36.660337 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:10:36.661623 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:36.651463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36191 (local address 127.0.97.2:58645)
0620 03:10:36.652305 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:36.652349 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:36.653977 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:36.654658 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:36.654667 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:36.654753 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:10:36.655278 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:36.655352 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:36.656363 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:36.656369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:36.657792 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:36.657804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:36.658613 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:36.658618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:36.658819 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:36.659527 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:10:36.659530 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:10:36.659572 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:10:36.659646 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:10:36.660016 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:10:36.660028 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:10:36.660037 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 03:10:36.660212 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 03:10:36.660599 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":274,"threads_started":1}
I20260620 03:10:36.662984 20584 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36191
I20260620 03:10:36.663331 20584 heartbeater.cc:464] Registering TS with master...
I20260620 03:10:36.664084 20584 heartbeater.cc:511] Master 127.0.97.62:36191 requested a full tablet report, sending...
I20260620 03:10:36.665572 20267 ts_manager.cc:194] Registered new tserver with Master: c9f6c5b1ed66455c8132abc6c83cb3fb (127.0.97.2:42523)
I20260620 03:10:36.666654 20267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:58645
W20260620 03:10:36.978932 20587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:36.979185 20587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:36.979269 20587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:36.986344 20587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:36.986465 20587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:36.986577 20587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:10:36.995499 20587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:36.996641 20587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:36.998286 20587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:37.011271 20592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:37.011339 20593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:37.011596 20595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:37.012853 20587 server_base.cc:1034] running on GCE node
I20260620 03:10:37.013434 20587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:37.014375 20587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:37.015643 20587 hybrid_clock.cc:648] HybridClock initialized: now 1781925037015586 us; error 48 us; skew 500 ppm
I20260620 03:10:37.015985 20587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:10:37.018321 20587 webserver.cc:460] Webserver started at http://127.0.97.3:34521/ using document root <none> and password file <none>
I20260620 03:10:37.018996 20587 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:37.019109 20587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:37.019507 20587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:37.022303 20587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-2/data/instance:
uuid: "ef62f998055c4ab28429d1d14da0980c"
format_stamp: "Formatted at 2026-06-20 03:10:37 on dist-test-slave-xv68"
server_key: "c2d7bceb8df480359ac5398d80dd7cbb"
server_key_iv: "3e6010d80fd651d73b1f347313579cf6"
server_key_version: "encryptionkey@0"
I20260620 03:10:37.023064 20587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ef62f998055c4ab28429d1d14da0980c"
format_stamp: "Formatted at 2026-06-20 03:10:37 on dist-test-slave-xv68"
server_key: "c2d7bceb8df480359ac5398d80dd7cbb"
server_key_iv: "3e6010d80fd651d73b1f347313579cf6"
server_key_version: "encryptionkey@0"
I20260620 03:10:37.028020 20587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 03:10:37.031185 20601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:37.032449 20587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 03:10:37.032612 20587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-2/wal
uuid: "ef62f998055c4ab28429d1d14da0980c"
format_stamp: "Formatted at 2026-06-20 03:10:37 on dist-test-slave-xv68"
server_key: "c2d7bceb8df480359ac5398d80dd7cbb"
server_key_iv: "3e6010d80fd651d73b1f347313579cf6"
server_key_version: "encryptionkey@0"
I20260620 03:10:37.032820 20587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:37.055168 20587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:37.055963 20587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:37.056241 20587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:37.057138 20587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:10:37.058542 20587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:10:37.058673 20587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:37.058797 20587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:10:37.058884 20587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:37.090806 20587 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33823
I20260620 03:10:37.090890 20713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33823 every 8 connection(s)
I20260620 03:10:37.092132 20587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:10:37.093895 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 20587
I20260620 03:10:37.094074 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:10:37.094660 388 external_mini_cluster.cc:1428] Setting key e8fd96c1a7deaa1fb0ef13a7aaf75691
I20260620 03:10:37.104435 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.094848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54705 (local address 127.0.97.62:36191)
0620 03:10:37.095036 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:37.095042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:37.096740 (+ 1698us) server_negotiation.cc:407] Connection header received
0620 03:10:37.098451 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:37.098454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:37.098517 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:37.098595 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:37.099727 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.100485 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.102150 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.102540 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.102628 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:37.103823 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:37.103828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:37.103831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:37.103847 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:37.103878 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:37.103889 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:37.103963 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:37.104147 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 03:10:37.104214 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 03:10:37.105373 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.095501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36191 (local address 127.0.97.3:54705)
0620 03:10:37.096581 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:37.096632 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:37.098221 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:37.098785 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:37.098792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:37.098873 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:10:37.099551 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:37.099578 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.100634 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.100638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:37.102032 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:37.102044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.102704 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.102708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:37.102885 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:37.103556 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:10:37.103558 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:10:37.103593 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:10:37.103659 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:10:37.103972 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:10:37.103980 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:10:37.103985 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 03:10:37.104108 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 03:10:37.104355 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":379,"threads_started":1}
I20260620 03:10:37.106516 20714 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36191
I20260620 03:10:37.106782 20714 heartbeater.cc:464] Registering TS with master...
I20260620 03:10:37.107563 20714 heartbeater.cc:511] Master 127.0.97.62:36191 requested a full tablet report, sending...
I20260620 03:10:37.109071 20267 ts_manager.cc:194] Registered new tserver with Master: ef62f998055c4ab28429d1d14da0980c (127.0.97.3:33823)
I20260620 03:10:37.109570 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:10:37.109967 20267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:54705
I20260620 03:10:37.118986 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.113569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.0.97.62:36191)
0620 03:10:37.113713 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:37.113717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:37.114185 (+ 468us) server_negotiation.cc:407] Connection header received
0620 03:10:37.114303 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:37.114306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:37.114369 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:37.114449 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:37.114874 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.115668 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.116951 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.117367 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.117458 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:37.118446 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:37.118450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:37.118454 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:37.118469 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:37.118498 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:37.118510 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:37.118580 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:37.118701 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 03:10:37.118767 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 03:10:37.122365 20267 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:10:37.128855 20267 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:10:37.167253 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.141895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38499 (local address 127.0.0.1:60412)
0620 03:10:37.158107 (+ 16212us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:37.158126 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:37.158231 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:37.159547 (+ 1316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:37.159560 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:37.159603 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:37.160012 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:37.160026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.162007 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.162013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:37.164540 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:37.164555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.166298 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.166304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:37.166484 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:37.166533 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 03:10:37.166676 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 03:10:37.166862 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16044,"mutex_wait_us":34,"thread_start_us":15894,"threads_started":1}
I20260620 03:10:37.167268 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.142727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60412 (local address 127.0.97.1:38499)
0620 03:10:37.158716 (+ 15989us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:37.158729 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:37.158768 (+ 39us) server_negotiation.cc:407] Connection header received
0620 03:10:37.158887 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:37.158898 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:37.159241 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:37.159520 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:37.160215 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.161850 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.164766 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.166119 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.166273 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:37.166530 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:37.166735 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 03:10:37.166913 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15833,"thread_start_us":15724,"threads_started":1}
I20260620 03:10:37.167591 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.141254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33823 (local address 127.0.0.1:33636)
0620 03:10:37.157455 (+ 16201us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:37.157477 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:37.157605 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:37.160649 (+ 3044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:37.160654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:37.160678 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:37.161085 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:37.161097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.162832 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.162841 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:37.164910 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:37.164926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.166671 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.166676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:37.166856 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:37.166886 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 03:10:37.166967 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 03:10:37.167078 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15539,"mutex_wait_us":85,"thread_start_us":866,"threads_started":1}
I20260620 03:10:37.168529 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.141928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.0.97.3:33823)
0620 03:10:37.160223 (+ 18295us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:37.160228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:37.160253 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:10:37.160324 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:37.160328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:37.160478 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:37.160658 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:37.161316 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.162693 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.165117 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.166177 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.166303 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:37.166564 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:37.168003 (+ 1439us) server_negotiation.cc:299] Negotiation successful
0620 03:10:37.168176 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18126,"thread_start_us":17983,"threads_started":1}
I20260620 03:10:37.168540 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.140909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42523 (local address 127.0.0.1:37990)
0620 03:10:37.155968 (+ 15059us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:37.156020 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:37.156220 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:37.160946 (+ 4726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:37.160952 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:37.160980 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:37.161359 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:37.161367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.163147 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.163152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:37.165702 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:37.165719 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.167937 (+ 2218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.167943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:37.168105 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:37.168128 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 03:10:37.168188 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 03:10:37.168313 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14728,"thread_start_us":15499,"threads_started":1}
I20260620 03:10:37.169433 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.141097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37990 (local address 127.0.97.2:42523)
0620 03:10:37.160249 (+ 19152us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:37.160258 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:37.160287 (+ 29us) server_negotiation.cc:407] Connection header received
0620 03:10:37.160382 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:37.160389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:37.160637 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:37.160815 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:37.161507 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.162959 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.165897 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.167352 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.167485 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:37.168878 (+ 1393us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:37.168955 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 03:10:37.169108 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18971,"thread_start_us":19901,"threads_started":1}
I20260620 03:10:37.171253 20388 tablet_service.cc:1467] Processing CreateTablet for tablet a21ea703d8434818bdf979e2d91aa111 (DEFAULT_TABLE table=test-table [id=bc633e188b414926bf6f8064f7bd0e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:10:37.171586 20649 tablet_service.cc:1467] Processing CreateTablet for tablet a21ea703d8434818bdf979e2d91aa111 (DEFAULT_TABLE table=test-table [id=bc633e188b414926bf6f8064f7bd0e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:10:37.172772 20388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21ea703d8434818bdf979e2d91aa111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:37.172952 20649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21ea703d8434818bdf979e2d91aa111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:37.174898 20519 tablet_service.cc:1467] Processing CreateTablet for tablet a21ea703d8434818bdf979e2d91aa111 (DEFAULT_TABLE table=test-table [id=bc633e188b414926bf6f8064f7bd0e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:10:37.176198 20519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21ea703d8434818bdf979e2d91aa111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:37.184038 20732 tablet_bootstrap.cc:492] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c: Bootstrap starting.
I20260620 03:10:37.187543 20732 tablet_bootstrap.cc:654] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:37.188772 20732 log.cc:824] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:37.199014 20733 tablet_bootstrap.cc:492] T a21ea703d8434818bdf979e2d91aa111 P ae366687b9634d25890b1f0eea4ab447: Bootstrap starting.
I20260620 03:10:37.200736 20732 tablet_bootstrap.cc:492] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c: No bootstrap required, opened a new log
I20260620 03:10:37.201037 20732 ts_tablet_manager.cc:1397] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.011s
I20260620 03:10:37.202719 20733 tablet_bootstrap.cc:654] T a21ea703d8434818bdf979e2d91aa111 P ae366687b9634d25890b1f0eea4ab447: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:37.204005 20733 log.cc:824] T a21ea703d8434818bdf979e2d91aa111 P ae366687b9634d25890b1f0eea4ab447: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:37.204754 20735 tablet_bootstrap.cc:492] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb: Bootstrap starting.
I20260620 03:10:37.208685 20735 tablet_bootstrap.cc:654] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:37.209908 20735 log.cc:824] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:37.210148 20733 tablet_bootstrap.cc:492] T a21ea703d8434818bdf979e2d91aa111 P ae366687b9634d25890b1f0eea4ab447: No bootstrap required, opened a new log
I20260620 03:10:37.210450 20733 ts_tablet_manager.cc:1397] T a21ea703d8434818bdf979e2d91aa111 P ae366687b9634d25890b1f0eea4ab447: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260620 03:10:37.210798 20453 heartbeater.cc:503] Master 127.0.97.62:36191 was elected leader, sending a full tablet report...
I20260620 03:10:37.214146 20732 raft_consensus.cc:348] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f6c5b1ed66455c8132abc6c83cb3fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42523 } } peers { permanent_uuid: "ef62f998055c4ab28429d1d14da0980c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33823 } } peers { permanent_uuid: "ae366687b9634d25890b1f0eea4ab447" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } }
I20260620 03:10:37.214532 20732 raft_consensus.cc:374] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:37.214627 20732 raft_consensus.cc:729] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef62f998055c4ab28429d1d14da0980c, State: Initialized, Role: FOLLOWER
I20260620 03:10:37.215305 20732 consensus_queue.cc:260] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f6c5b1ed66455c8132abc6c83cb3fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42523 } } peers { permanent_uuid: "ef62f998055c4ab28429d1d14da0980c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33823 } } peers { permanent_uuid: "ae366687b9634d25890b1f0eea4ab447" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } }
I20260620 03:10:37.216418 20714 heartbeater.cc:503] Master 127.0.97.62:36191 was elected leader, sending a full tablet report...
I20260620 03:10:37.216775 20735 tablet_bootstrap.cc:492] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb: No bootstrap required, opened a new log
I20260620 03:10:37.217096 20735 ts_tablet_manager.cc:1397] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.006s
I20260620 03:10:37.217120 20732 ts_tablet_manager.cc:1428] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
I20260620 03:10:37.221001 20733 raft_consensus.cc:348] T a21ea703d8434818bdf979e2d91aa111 P ae366687b9634d25890b1f0eea4ab447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f6c5b1ed66455c8132abc6c83cb3fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42523 } } peers { permanent_uuid: "ef62f998055c4ab28429d1d14da0980c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33823 } } peers { permanent_uuid: "ae366687b9634d25890b1f0eea4ab447" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } }
I20260620 03:10:37.221446 20733 raft_consensus.cc:374] T a21ea703d8434818bdf979e2d91aa111 P ae366687b9634d25890b1f0eea4ab447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:37.221546 20733 raft_consensus.cc:729] T a21ea703d8434818bdf979e2d91aa111 P ae366687b9634d25890b1f0eea4ab447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae366687b9634d25890b1f0eea4ab447, State: Initialized, Role: FOLLOWER
I20260620 03:10:37.222123 20733 consensus_queue.cc:260] T a21ea703d8434818bdf979e2d91aa111 P ae366687b9634d25890b1f0eea4ab447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f6c5b1ed66455c8132abc6c83cb3fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42523 } } peers { permanent_uuid: "ef62f998055c4ab28429d1d14da0980c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33823 } } peers { permanent_uuid: "ae366687b9634d25890b1f0eea4ab447" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } }
I20260620 03:10:37.223846 20733 ts_tablet_manager.cc:1428] T a21ea703d8434818bdf979e2d91aa111 P ae366687b9634d25890b1f0eea4ab447: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
I20260620 03:10:37.226227 20735 raft_consensus.cc:348] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f6c5b1ed66455c8132abc6c83cb3fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42523 } } peers { permanent_uuid: "ef62f998055c4ab28429d1d14da0980c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33823 } } peers { permanent_uuid: "ae366687b9634d25890b1f0eea4ab447" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } }
I20260620 03:10:37.226572 20735 raft_consensus.cc:374] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:37.226686 20735 raft_consensus.cc:729] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f6c5b1ed66455c8132abc6c83cb3fb, State: Initialized, Role: FOLLOWER
I20260620 03:10:37.227195 20735 consensus_queue.cc:260] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f6c5b1ed66455c8132abc6c83cb3fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42523 } } peers { permanent_uuid: "ef62f998055c4ab28429d1d14da0980c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33823 } } peers { permanent_uuid: "ae366687b9634d25890b1f0eea4ab447" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } }
I20260620 03:10:37.228240 20584 heartbeater.cc:503] Master 127.0.97.62:36191 was elected leader, sending a full tablet report...
I20260620 03:10:37.228654 20735 ts_tablet_manager.cc:1428] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s
I20260620 03:10:37.313650 20740 raft_consensus.cc:482] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:10:37.313989 20740 raft_consensus.cc:504] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f6c5b1ed66455c8132abc6c83cb3fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42523 } } peers { permanent_uuid: "ef62f998055c4ab28429d1d14da0980c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33823 } } peers { permanent_uuid: "ae366687b9634d25890b1f0eea4ab447" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } }
I20260620 03:10:37.315755 20740 leader_election.cc:288] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef62f998055c4ab28429d1d14da0980c (127.0.97.3:33823), ae366687b9634d25890b1f0eea4ab447 (127.0.97.1:38499)
I20260620 03:10:37.322604 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.316185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34089 (local address 127.0.97.1:38499)
0620 03:10:37.316427 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:37.316434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:37.317196 (+ 762us) server_negotiation.cc:407] Connection header received
0620 03:10:37.317364 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:37.317370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:37.317525 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:37.317623 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:37.318230 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.318898 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.320930 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.321662 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.321757 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:37.321861 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:37.322221 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 03:10:37.322354 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 03:10:37.322614 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.316043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52119 (local address 127.0.97.3:33823)
0620 03:10:37.316274 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:37.316284 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:37.317067 (+ 783us) server_negotiation.cc:407] Connection header received
0620 03:10:37.317296 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:37.317300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:37.317391 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:37.317478 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:10:37.318116 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.318857 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.320875 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.321737 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.321868 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:37.321973 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:37.322276 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 03:10:37.322395 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:10:37.322695 20741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.315974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33823 (local address 127.0.97.2:52119)
0620 03:10:37.316955 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:37.316970 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:37.317145 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:37.317514 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:37.317518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:37.317540 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:37.317966 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:37.317975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.318984 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.318992 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:37.320676 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:37.320687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.321930 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.321935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:37.322102 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:37.322124 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 03:10:37.322236 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 03:10:37.322351 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":96,"thread_start_us":641,"threads_started":1}
I20260620 03:10:37.323180 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.316040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38499 (local address 127.0.97.2:34089)
0620 03:10:37.317048 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:37.317063 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:10:37.317209 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:10:37.317649 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:10:37.317652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:10:37.317679 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:10:37.318071 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:37.318083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.319057 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.319061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:37.320682 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:10:37.320693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.321853 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.321860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:10:37.322063 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:37.322088 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 03:10:37.322156 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 03:10:37.322284 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":45,"thread_start_us":600,"threads_started":1}
I20260620 03:10:37.323768 20669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21ea703d8434818bdf979e2d91aa111" candidate_uuid: "c9f6c5b1ed66455c8132abc6c83cb3fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef62f998055c4ab28429d1d14da0980c" is_pre_election: true
I20260620 03:10:37.324275 20408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21ea703d8434818bdf979e2d91aa111" candidate_uuid: "c9f6c5b1ed66455c8132abc6c83cb3fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae366687b9634d25890b1f0eea4ab447" is_pre_election: true
I20260620 03:10:37.324450 20669 raft_consensus.cc:2454] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f6c5b1ed66455c8132abc6c83cb3fb in term 0.
I20260620 03:10:37.324797 20408 raft_consensus.cc:2454] T a21ea703d8434818bdf979e2d91aa111 P ae366687b9634d25890b1f0eea4ab447 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f6c5b1ed66455c8132abc6c83cb3fb in term 0.
I20260620 03:10:37.325170 20475 leader_election.cc:302] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f6c5b1ed66455c8132abc6c83cb3fb, ef62f998055c4ab28429d1d14da0980c; no voters:
I20260620 03:10:37.325543 20740 raft_consensus.cc:2781] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:10:37.325687 20740 raft_consensus.cc:482] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:10:37.325795 20740 raft_consensus.cc:3037] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:37.327991 20740 raft_consensus.cc:504] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f6c5b1ed66455c8132abc6c83cb3fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42523 } } peers { permanent_uuid: "ef62f998055c4ab28429d1d14da0980c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33823 } } peers { permanent_uuid: "ae366687b9634d25890b1f0eea4ab447" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } }
I20260620 03:10:37.328581 20740 leader_election.cc:288] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [CANDIDATE]: Term 1 election: Requested vote from peers ef62f998055c4ab28429d1d14da0980c (127.0.97.3:33823), ae366687b9634d25890b1f0eea4ab447 (127.0.97.1:38499)
I20260620 03:10:37.329011 20669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21ea703d8434818bdf979e2d91aa111" candidate_uuid: "c9f6c5b1ed66455c8132abc6c83cb3fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef62f998055c4ab28429d1d14da0980c"
I20260620 03:10:37.329103 20408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21ea703d8434818bdf979e2d91aa111" candidate_uuid: "c9f6c5b1ed66455c8132abc6c83cb3fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae366687b9634d25890b1f0eea4ab447"
I20260620 03:10:37.329213 20669 raft_consensus.cc:3037] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:37.329340 20408 raft_consensus.cc:3037] T a21ea703d8434818bdf979e2d91aa111 P ae366687b9634d25890b1f0eea4ab447 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:37.331980 20669 raft_consensus.cc:2454] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f6c5b1ed66455c8132abc6c83cb3fb in term 1.
I20260620 03:10:37.332063 20408 raft_consensus.cc:2454] T a21ea703d8434818bdf979e2d91aa111 P ae366687b9634d25890b1f0eea4ab447 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f6c5b1ed66455c8132abc6c83cb3fb in term 1.
I20260620 03:10:37.332376 20475 leader_election.cc:302] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f6c5b1ed66455c8132abc6c83cb3fb, ef62f998055c4ab28429d1d14da0980c; no voters:
I20260620 03:10:37.332675 20740 raft_consensus.cc:2781] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:37.333129 20740 raft_consensus.cc:686] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [term 1 LEADER]: Becoming Leader. State: Replica: c9f6c5b1ed66455c8132abc6c83cb3fb, State: Running, Role: LEADER
I20260620 03:10:37.333598 20740 consensus_queue.cc:237] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f6c5b1ed66455c8132abc6c83cb3fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42523 } } peers { permanent_uuid: "ef62f998055c4ab28429d1d14da0980c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33823 } } peers { permanent_uuid: "ae366687b9634d25890b1f0eea4ab447" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } }
I20260620 03:10:37.338590 20267 catalog_manager.cc:5495] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f6c5b1ed66455c8132abc6c83cb3fb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c9f6c5b1ed66455c8132abc6c83cb3fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f6c5b1ed66455c8132abc6c83cb3fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef62f998055c4ab28429d1d14da0980c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae366687b9634d25890b1f0eea4ab447" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } health_report { overall_health: UNKNOWN } } }
W20260620 03:10:37.343786 20715 tablet.cc:2369] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:10:37.388613 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.383507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37994 (local address 127.0.97.2:42523)
0620 03:10:37.383691 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:37.383696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:37.383713 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:10:37.383867 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:37.383871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:37.383958 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:37.384082 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:10:37.384432 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.385180 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.386508 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.386918 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.387011 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:37.387600 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:10:37.388168 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:10:37.388292 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:37.388358 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 03:10:37.388421 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:10:37.397207 20408 raft_consensus.cc:1261] T a21ea703d8434818bdf979e2d91aa111 P ae366687b9634d25890b1f0eea4ab447 [term 1 FOLLOWER]: Refusing update from remote peer c9f6c5b1ed66455c8132abc6c83cb3fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:10:37.397229 20669 raft_consensus.cc:1261] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c [term 1 FOLLOWER]: Refusing update from remote peer c9f6c5b1ed66455c8132abc6c83cb3fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:10:37.398319 20740 consensus_queue.cc:1035] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef62f998055c4ab28429d1d14da0980c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:10:37.398648 20743 consensus_queue.cc:1035] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae366687b9634d25890b1f0eea4ab447" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 03:10:37.403582 20585 tablet.cc:2369] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:10:37.412158 20746 mvcc.cc:205] Tried to move back new op lower bound from 7298764953166987264 to 7298764952922411008. Current Snapshot: MvccSnapshot[applied={T|T < 7298764953166987264}]
I20260620 03:10:37.417382 20747 mvcc.cc:205] Tried to move back new op lower bound from 7298764953166987264 to 7298764952922411008. Current Snapshot: MvccSnapshot[applied={T|T < 7298764953166987264}]
I20260620 03:10:37.418135 20748 mvcc.cc:205] Tried to move back new op lower bound from 7298764953166987264 to 7298764952922411008. Current Snapshot: MvccSnapshot[applied={T|T < 7298764953166987264}]
I20260620 03:10:37.427488 20266 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:10:37.427793 20266 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:10:37.431643 20266 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca654b78377049ffb8bea8ddfe1c9329: Sending DeleteTablet for 3 replicas of tablet a21ea703d8434818bdf979e2d91aa111
I20260620 03:10:37.432760 20519 tablet_service.cc:1514] Processing DeleteTablet for tablet a21ea703d8434818bdf979e2d91aa111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:10:37 UTC) from {username='slave'} at 127.0.0.1:37990
I20260620 03:10:37.432744 20649 tablet_service.cc:1514] Processing DeleteTablet for tablet a21ea703d8434818bdf979e2d91aa111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:10:37 UTC) from {username='slave'} at 127.0.0.1:33636
I20260620 03:10:37.432817 20388 tablet_service.cc:1514] Processing DeleteTablet for tablet a21ea703d8434818bdf979e2d91aa111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:10:37 UTC) from {username='slave'} at 127.0.0.1:60412
I20260620 03:10:37.435066 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 20326
I20260620 03:10:37.445375 20756 tablet_replica.cc:330] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb: stopping tablet replica
I20260620 03:10:37.445804 20756 raft_consensus.cc:2229] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:10:37.446535 20756 raft_consensus.cc:2258] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:10:37.450377 20757 tablet_replica.cc:330] T a21ea703d8434818bdf979e2d91aa111 P ae366687b9634d25890b1f0eea4ab447: stopping tablet replica
I20260620 03:10:37.450789 20756 ts_tablet_manager.cc:1905] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:10:37.450830 20757 raft_consensus.cc:2229] T a21ea703d8434818bdf979e2d91aa111 P ae366687b9634d25890b1f0eea4ab447 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:10:37.451335 20757 raft_consensus.cc:2258] T a21ea703d8434818bdf979e2d91aa111 P ae366687b9634d25890b1f0eea4ab447 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:10:37.452912 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.436719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60428 (local address 127.0.97.1:38499)
0620 03:10:37.436977 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:37.436986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:37.441325 (+ 4339us) server_negotiation.cc:407] Connection header received
0620 03:10:37.442186 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:37.442195 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:37.442341 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:37.442450 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:37.443754 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.445321 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.448002 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.448963 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.449085 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:37.450799 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:37.450811 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:37.450832 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:37.451769 (+ 937us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:37.451842 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:37.451873 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:37.451998 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:37.452266 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 03:10:37.452430 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 03:10:37.454723 20448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:37.455848 20755 tablet_replica.cc:330] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c: stopping tablet replica
I20260620 03:10:37.456246 20755 raft_consensus.cc:2229] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:10:37.456724 20755 raft_consensus.cc:2258] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:10:37.458812 20755 ts_tablet_manager.cc:1905] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:10:37.460562 20756 ts_tablet_manager.cc:1918] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:10:37.460775 20756 log.cc:1196] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-1/wal/wals/a21ea703d8434818bdf979e2d91aa111
I20260620 03:10:37.461459 20756 ts_tablet_manager.cc:1939] T a21ea703d8434818bdf979e2d91aa111 P c9f6c5b1ed66455c8132abc6c83cb3fb: Deleting consensus metadata
I20260620 03:10:37.463742 20251 catalog_manager.cc:4841] TS c9f6c5b1ed66455c8132abc6c83cb3fb (127.0.97.2:42523): tablet a21ea703d8434818bdf979e2d91aa111 (table test-table [id=bc633e188b414926bf6f8064f7bd0e3a]) successfully deleted
I20260620 03:10:37.466012 20755 ts_tablet_manager.cc:1918] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:10:37.466190 20755 log.cc:1196] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c: Deleting WAL directory at /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924720365844-388-0/minicluster-data/ts-2/wal/wals/a21ea703d8434818bdf979e2d91aa111
I20260620 03:10:37.466737 20755 ts_tablet_manager.cc:1939] T a21ea703d8434818bdf979e2d91aa111 P ef62f998055c4ab28429d1d14da0980c: Deleting consensus metadata
I20260620 03:10:37.468667 20254 catalog_manager.cc:4841] TS ef62f998055c4ab28429d1d14da0980c (127.0.97.3:33823): tablet a21ea703d8434818bdf979e2d91aa111 (table test-table [id=bc633e188b414926bf6f8064f7bd0e3a]) successfully deleted
I20260620 03:10:37.455634 20757 ts_tablet_manager.cc:1905] T a21ea703d8434818bdf979e2d91aa111 P ae366687b9634d25890b1f0eea4ab447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:10:37.553009 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 20326
W20260620 03:10:37.570878 20252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38499 (error 108)
I20260620 03:10:37.571031 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 20457
I20260620 03:10:37.572894 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.572449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38499 (local address 127.0.0.1:45640)
0620 03:10:37.572645 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:37.572757 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 03:10:37.573251 20252 catalog_manager.cc:4584] TS ae366687b9634d25890b1f0eea4ab447 (127.0.97.1:38499): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a21ea703d8434818bdf979e2d91aa111: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38499: connect: Connection refused (error 111)
I20260620 03:10:37.579609 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.571865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40748 (local address 127.0.97.2:42523)
0620 03:10:37.572057 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:37.572065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:37.572321 (+ 256us) server_negotiation.cc:407] Connection header received
0620 03:10:37.572395 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:37.572401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:37.572504 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:37.572600 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:37.573267 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.574132 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.576236 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.576735 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.576838 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:37.578118 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:37.578127 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:37.578141 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:37.578756 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:37.578810 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:37.578832 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:37.578921 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:37.579094 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 03:10:37.579177 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 03:10:37.580281 20579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:37.626847 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.626413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38499 (local address 127.0.0.1:45648)
0620 03:10:37.626612 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:37.626693 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 03:10:37.627831 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.627493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38499 (local address 127.0.0.1:45662)
0620 03:10:37.627650 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:37.627714 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 03:10:37.688329 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.687810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38499 (local address 127.0.0.1:45676)
0620 03:10:37.688019 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:37.688161 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 03:10:37.689316 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.688973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38499 (local address 127.0.0.1:45678)
0620 03:10:37.689125 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:37.689209 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 03:10:37.705449 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 20457
I20260620 03:10:37.724491 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 20587
I20260620 03:10:37.735743 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.725230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46122 (local address 127.0.97.3:33823)
0620 03:10:37.725429 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:37.725434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:37.725452 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:10:37.725644 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:37.725648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:37.725735 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:37.725853 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:37.726221 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.726939 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.728310 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:37.728705 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:37.728810 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:37.734522 (+ 5712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:37.734531 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:37.734546 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:37.735098 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:37.735146 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:37.735165 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:37.735250 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:37.735395 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 03:10:37.735489 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 03:10:37.736413 20709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:37.770102 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.769576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38499 (local address 127.0.0.1:45692)
0620 03:10:37.769798 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:37.769905 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 03:10:37.771159 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.770729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38499 (local address 127.0.0.1:45706)
0620 03:10:37.770929 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:37.771023 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 03:10:37.824596 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 20587
I20260620 03:10:37.841194 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 20236
I20260620 03:10:37.842264 20297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:37.954382 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 20236
I20260620 03:10:37.954499 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:37.953760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38499 (local address 127.0.0.1:45708)
0620 03:10:37.954226 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 03:10:37.954340 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283}
2026-06-20T03:10:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2823 ms)
[----------] 6 tests from EncryptionPolicyTest (21218 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:10:38 dist-test-slave-xv68 krb5kdc[20766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:10:38 dist-test-slave-xv68 krb5kdc[20766](info): set up 2 sockets
Jun 20 03:10:38 dist-test-slave-xv68 krb5kdc[20766](info): commencing operation
krb5kdc: starting...
W20260620 03:10:39.071784 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:10:39 dist-test-slave-xv68 krb5kdc[20766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:10:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:10:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:10:39.314010 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:39.635532 20779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:39.635790 20779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:39.636024 20779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:39.643118 20779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:10:39.643244 20779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:39.643352 20779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:39.643483 20779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:10:39.643559 20779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:10:39.653169 20779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32945
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:39.654390 20779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:39.655725 20779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:39.667485 20784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:39.667594 20787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:39.667522 20785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:39.668475 20779 server_base.cc:1034] running on GCE node
I20260620 03:10:39.669180 20779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:39.670492 20779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:39.671712 20779 hybrid_clock.cc:648] HybridClock initialized: now 1781925039671656 us; error 47 us; skew 500 ppm
I20260620 03:10:39.672101 20779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:39 dist-test-slave-xv68 krb5kdc[20766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:39.675117 20779 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:10:39.676617 20779 webserver.cc:460] Webserver started at http://127.0.0.1:43783/ using document root <none> and password file <none>
I20260620 03:10:39.677278 20779 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:39.677382 20779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:39.677680 20779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:39.680714 20779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "c975b62f73c04c39861d978ef06ce88a"
format_stamp: "Formatted at 2026-06-20 03:10:39 on dist-test-slave-xv68"
server_key: "503db5a784c4b2ad481c8d92b421bcb6"
server_key_iv: "44bfe8972c98ef19f08d6b7c1c5fe276"
server_key_version: "encryptionkey@0"
I20260620 03:10:39.681507 20779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "c975b62f73c04c39861d978ef06ce88a"
format_stamp: "Formatted at 2026-06-20 03:10:39 on dist-test-slave-xv68"
server_key: "503db5a784c4b2ad481c8d92b421bcb6"
server_key_iv: "44bfe8972c98ef19f08d6b7c1c5fe276"
server_key_version: "encryptionkey@0"
I20260620 03:10:39.686571 20779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 03:10:39.689942 20794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:39.691246 20779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:10:39.691501 20779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "c975b62f73c04c39861d978ef06ce88a"
format_stamp: "Formatted at 2026-06-20 03:10:39 on dist-test-slave-xv68"
server_key: "503db5a784c4b2ad481c8d92b421bcb6"
server_key_iv: "44bfe8972c98ef19f08d6b7c1c5fe276"
server_key_version: "encryptionkey@0"
I20260620 03:10:39.691696 20779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:39.708148 20779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:39.712062 20779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:39.712361 20779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:39.730613 20779 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32945
I20260620 03:10:39.730646 20845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32945 every 8 connection(s)
I20260620 03:10:39.732036 20779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:10:39.735425 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 20779
I20260620 03:10:39.735648 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:10:39.735917 20846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:39.736483 388 external_mini_cluster.cc:1428] Setting key 7a179f8daeee98876236a7b89e0b969c
I20260620 03:10:39.750284 20846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a: Bootstrap starting.
Jun 20 03:10:39 dist-test-slave-xv68 krb5kdc[20766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 03:10:39.754027 20846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:39.755342 20846 log.cc:824] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:39.758641 20846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a: No bootstrap required, opened a new log
I20260620 03:10:39.760447 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:39.738339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47661 (local address 127.0.0.1:32945)
0620 03:10:39.739152 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:39.739163 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:39.739203 (+ 40us) server_negotiation.cc:407] Connection header received
0620 03:10:39.740940 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:39.740960 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:39.741324 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:39.741643 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:39.743422 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:39.744632 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:39.746456 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:39.747126 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:39.747267 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:39.751949 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:39.751961 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:39.751995 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:39.752049 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:39.755826 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:39.756845 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:39.756855 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:39.756862 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:39.757063 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:39.757478 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:39.757485 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:39.757490 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:39.758180 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:39.758491 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:39.758848 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 03:10:39.759189 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":386,"threads_started":1}
I20260620 03:10:39.764168 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:10:39.768160 20846 raft_consensus.cc:348] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c975b62f73c04c39861d978ef06ce88a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32945 } }
I20260620 03:10:39.768625 20846 raft_consensus.cc:374] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:39.768729 20846 raft_consensus.cc:729] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c975b62f73c04c39861d978ef06ce88a, State: Initialized, Role: FOLLOWER
I20260620 03:10:39.769343 20846 consensus_queue.cc:260] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c975b62f73c04c39861d978ef06ce88a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32945 } }
I20260620 03:10:39.769652 20846 raft_consensus.cc:388] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:10:39.769840 20846 raft_consensus.cc:482] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:10:39.770081 20846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:39.772662 20846 raft_consensus.cc:504] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c975b62f73c04c39861d978ef06ce88a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32945 } }
I20260620 03:10:39.773201 20846 leader_election.cc:302] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c975b62f73c04c39861d978ef06ce88a; no voters:
I20260620 03:10:39.773883 20846 leader_election.cc:288] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:10:39.773988 20856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:39.775206 20856 raft_consensus.cc:686] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a [term 1 LEADER]: Becoming Leader. State: Replica: c975b62f73c04c39861d978ef06ce88a, State: Running, Role: LEADER
I20260620 03:10:39.775856 20856 consensus_queue.cc:237] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c975b62f73c04c39861d978ef06ce88a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32945 } }
I20260620 03:10:39.776166 20846 sys_catalog.cc:564] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:10:39.778826 20857 sys_catalog.cc:455] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c975b62f73c04c39861d978ef06ce88a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c975b62f73c04c39861d978ef06ce88a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32945 } } }
I20260620 03:10:39.779279 20857 sys_catalog.cc:458] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:39.783895 20858 sys_catalog.cc:455] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c975b62f73c04c39861d978ef06ce88a. Latest consensus state: current_term: 1 leader_uuid: "c975b62f73c04c39861d978ef06ce88a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c975b62f73c04c39861d978ef06ce88a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32945 } } }
I20260620 03:10:39.784221 20858 sys_catalog.cc:458] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:39.785123 20861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:10:39.788184 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:39.769778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58843 (local address 127.0.0.1:32945)
0620 03:10:39.770010 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:39.770016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:39.770417 (+ 401us) server_negotiation.cc:407] Connection header received
0620 03:10:39.770527 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:39.770532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:39.770607 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:39.770725 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:39.772135 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:39.773257 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:39.774950 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:39.775634 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:39.775769 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:39.778783 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:39.778788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:39.778793 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:39.778831 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:39.785201 (+ 6370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:39.786081 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:39.786087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:39.786094 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:39.786212 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:39.786645 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:39.786650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:39.786653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:39.786983 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:39.787282 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:39.787644 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 03:10:39.787818 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 03:10:39.795826 20861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:10:39.805923 20861 catalog_manager.cc:1347] Generated new cluster ID: c3e8ca331ff541b3a4d8c811972b8e78
I20260620 03:10:39.806058 20861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:10:39.827903 20861 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:10:39.829698 20861 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:10:39.837778 20861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c975b62f73c04c39861d978ef06ce88a: Generated new TSK 0
I20260620 03:10:39.838719 20861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:10:39.843998 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 20779
I20260620 03:10:39.850487 20841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:39.942893 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 20779
W20260620 03:10:39.960506 20764 connection.cc:535] client connection to 127.0.0.1:32945 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32945): Connection reset by peer (error 104)
2026-06-20T03:10:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1983 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:10:40 dist-test-slave-xv68 krb5kdc[20878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:10:40 dist-test-slave-xv68 krb5kdc[20878](info): set up 2 sockets
Jun 20 03:10:40 dist-test-slave-xv68 krb5kdc[20878](info): commencing operation
krb5kdc: starting...
W20260620 03:10:41.043637 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.009s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:10:41 dist-test-slave-xv68 krb5kdc[20878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:10:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:10:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:10:41.288620 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33767
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:41.614099 20891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:41.614364 20891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:41.614555 20891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:41.621673 20891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:10:41.621798 20891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:41.621904 20891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:41.622009 20891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:10:41.622092 20891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:10:41.631698 20891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33767
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:41.632936 20891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:41.634613 20891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:41.646472 20896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:41.646420 20897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:41.646502 20899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:41.647337 20891 server_base.cc:1034] running on GCE node
I20260620 03:10:41.648099 20891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:41.649397 20891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:41.650617 20891 hybrid_clock.cc:648] HybridClock initialized: now 1781925041650529 us; error 80 us; skew 500 ppm
I20260620 03:10:41.650941 20891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:41 dist-test-slave-xv68 krb5kdc[20878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:41.654014 20891 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:10:41.655560 20891 webserver.cc:460] Webserver started at http://127.0.0.1:40427/ using document root <none> and password file <none>
I20260620 03:10:41.656214 20891 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:41.656358 20891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:41.656665 20891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:41.659677 20891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "86d797e007d7464984aae9bf09e296a2"
format_stamp: "Formatted at 2026-06-20 03:10:41 on dist-test-slave-xv68"
server_key: "9305b70271607ed562d3016d8c7f1018"
server_key_iv: "eee7f76f356b2d5459b8bc35e5b5f5da"
server_key_version: "encryptionkey@0"
I20260620 03:10:41.660461 20891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "86d797e007d7464984aae9bf09e296a2"
format_stamp: "Formatted at 2026-06-20 03:10:41 on dist-test-slave-xv68"
server_key: "9305b70271607ed562d3016d8c7f1018"
server_key_iv: "eee7f76f356b2d5459b8bc35e5b5f5da"
server_key_version: "encryptionkey@0"
I20260620 03:10:41.665977 20891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:10:41.669513 20906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:41.670832 20891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:10:41.671025 20891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "86d797e007d7464984aae9bf09e296a2"
format_stamp: "Formatted at 2026-06-20 03:10:41 on dist-test-slave-xv68"
server_key: "9305b70271607ed562d3016d8c7f1018"
server_key_iv: "eee7f76f356b2d5459b8bc35e5b5f5da"
server_key_version: "encryptionkey@0"
I20260620 03:10:41.671226 20891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:41.687839 20891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:41.692233 20891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:41.692523 20891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:41.711505 20891 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33767
I20260620 03:10:41.711525 20957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33767 every 8 connection(s)
I20260620 03:10:41.712852 20891 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:10:41.716552 20958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:41.721640 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 20891
I20260620 03:10:41.721848 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:10:41.722378 388 external_mini_cluster.cc:1428] Setting key b92f9d285b4a54ff48f92b47a6553a32
I20260620 03:10:41.727893 20958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2: Bootstrap starting.
I20260620 03:10:41.730918 20958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:41.731976 20958 log.cc:824] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:41.734987 20958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2: No bootstrap required, opened a new log
I20260620 03:10:41.735527 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:41.724163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37967 (local address 127.0.0.1:33767)
0620 03:10:41.725002 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:41.725014 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:41.725055 (+ 41us) server_negotiation.cc:407] Connection header received
0620 03:10:41.726813 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:41.726837 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:41.727205 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:41.727660 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:41.728197 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:41.729496 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:41.730831 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:41.731624 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:41.731824 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:41.733177 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:41.733190 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:41.733209 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:41.733737 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:41.733808 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:41.733832 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:41.733950 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:41.734272 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 03:10:41.734478 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":379,"threads_started":1}
I20260620 03:10:41.739516 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:10:41.743265 20958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d797e007d7464984aae9bf09e296a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33767 } }
I20260620 03:10:41.743937 20958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:41.744134 20958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d797e007d7464984aae9bf09e296a2, State: Initialized, Role: FOLLOWER
I20260620 03:10:41.744830 20958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d797e007d7464984aae9bf09e296a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33767 } }
I20260620 03:10:41.745133 20958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:10:41.745286 20958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:10:41.745502 20958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:41.748790 20958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d797e007d7464984aae9bf09e296a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33767 } }
I20260620 03:10:41.749429 20958 leader_election.cc:302] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86d797e007d7464984aae9bf09e296a2; no voters:
I20260620 03:10:41.750149 20958 leader_election.cc:288] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:10:41.750273 20968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:41.751401 20968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2 [term 1 LEADER]: Becoming Leader. State: Replica: 86d797e007d7464984aae9bf09e296a2, State: Running, Role: LEADER
I20260620 03:10:41.751748 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:41.743838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34447 (local address 127.0.0.1:33767)
0620 03:10:41.744062 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:41.744071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:41.744498 (+ 427us) server_negotiation.cc:407] Connection header received
0620 03:10:41.744684 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:41.744688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:41.744746 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:41.744832 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:41.745483 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:41.746317 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:41.748319 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:41.748754 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:41.748848 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:41.750225 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:41.750233 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:41.750241 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:41.750268 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:41.750316 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:41.750338 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:41.750487 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:41.751094 (+ 607us) server_negotiation.cc:299] Negotiation successful
0620 03:10:41.751164 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":59}
I20260620 03:10:41.752115 20968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d797e007d7464984aae9bf09e296a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33767 } }
I20260620 03:10:41.752969 20958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:10:41.755034 20970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86d797e007d7464984aae9bf09e296a2. Latest consensus state: current_term: 1 leader_uuid: "86d797e007d7464984aae9bf09e296a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d797e007d7464984aae9bf09e296a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33767 } } }
I20260620 03:10:41.755003 20969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86d797e007d7464984aae9bf09e296a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d797e007d7464984aae9bf09e296a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33767 } } }
I20260620 03:10:41.755496 20970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:41.755555 20969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:41.757105 20975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:10:41.762519 20975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:10:41.771570 20975 catalog_manager.cc:1347] Generated new cluster ID: 4df45fc8ef6846c6bbe336cf8cc88690
I20260620 03:10:41.771683 20975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:10:41.783751 20975 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:10:41.785568 20975 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:10:41.800048 20975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86d797e007d7464984aae9bf09e296a2: Generated new TSK 0
I20260620 03:10:41.801110 20975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:10:41.813247 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 20891
W20260620 03:10:41.814468 20953 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37967
W20260620 03:10:41.815214 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 20891: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:10:41.815412 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 20891
W20260620 03:10:41.832621 20876 connection.cc:535] client connection to 127.0.0.1:33767 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33767): Connection reset by peer (error 104)
2026-06-20T03:10:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1871 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:10:41 dist-test-slave-xv68 krb5kdc[20989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:10:41 dist-test-slave-xv68 krb5kdc[20989](info): set up 2 sockets
Jun 20 03:10:41 dist-test-slave-xv68 krb5kdc[20989](info): commencing operation
krb5kdc: starting...
W20260620 03:10:42.919678 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:10:43 dist-test-slave-xv68 krb5kdc[20989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:10:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:10:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:10:43.157807 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:43.479557 21002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:43.479874 21002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:43.480041 21002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:43.487197 21002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:10:43.487387 21002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:43.487488 21002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:43.487600 21002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:10:43.487684 21002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:10:43.497175 21002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34497
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:43.498456 21002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:43.499701 21002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:43.511045 21007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:43.510951 21008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:43.510980 21010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:43.511987 21002 server_base.cc:1034] running on GCE node
I20260620 03:10:43.512712 21002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:43.514003 21002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:43.515224 21002 hybrid_clock.cc:648] HybridClock initialized: now 1781925043515182 us; error 53 us; skew 500 ppm
I20260620 03:10:43.515579 21002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:43 dist-test-slave-xv68 krb5kdc[20989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:43.521750 21002 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:10:43.523066 21002 webserver.cc:460] Webserver started at http://127.0.0.1:44243/ using document root <none> and password file <none>
I20260620 03:10:43.523732 21002 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:43.523833 21002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:43.524089 21002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:43.526818 21002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "164535852e7a41208730603159995ed4"
format_stamp: "Formatted at 2026-06-20 03:10:43 on dist-test-slave-xv68"
server_key: "16f67d25c09f0b7b1fa62121bfa13ca5"
server_key_iv: "0746199e398dfb8964e8463ad5af8569"
server_key_version: "encryptionkey@0"
I20260620 03:10:43.527577 21002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "164535852e7a41208730603159995ed4"
format_stamp: "Formatted at 2026-06-20 03:10:43 on dist-test-slave-xv68"
server_key: "16f67d25c09f0b7b1fa62121bfa13ca5"
server_key_iv: "0746199e398dfb8964e8463ad5af8569"
server_key_version: "encryptionkey@0"
I20260620 03:10:43.532335 21002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 03:10:43.535468 21017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:43.536715 21002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:10:43.536886 21002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "164535852e7a41208730603159995ed4"
format_stamp: "Formatted at 2026-06-20 03:10:43 on dist-test-slave-xv68"
server_key: "16f67d25c09f0b7b1fa62121bfa13ca5"
server_key_iv: "0746199e398dfb8964e8463ad5af8569"
server_key_version: "encryptionkey@0"
I20260620 03:10:43.537072 21002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:43.542585 21002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:43.546839 21002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:43.547169 21002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:43.565546 21002 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34497
I20260620 03:10:43.565623 21068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34497 every 8 connection(s)
I20260620 03:10:43.566891 21002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:10:43.569271 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 21002
I20260620 03:10:43.569466 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:10:43.569926 388 external_mini_cluster.cc:1428] Setting key 3cdc570feab52151358c0b0b958b168f
I20260620 03:10:43.570799 21069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:43.578377 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:43.571709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56649 (local address 127.0.0.1:34497)
0620 03:10:43.572534 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:43.572549 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:43.572597 (+ 48us) server_negotiation.cc:407] Connection header received
0620 03:10:43.574439 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:43.574466 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:43.574915 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:43.575397 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:43.576365 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:43.576374 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:43.576390 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:43.576408 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:43.576507 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:43.576531 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:43.576744 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:43.576872 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 03:10:43.577064 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":318,"threads_started":1}
I20260620 03:10:43.582028 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:10:43.585265 21069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4: Bootstrap starting.
I20260620 03:10:43.588774 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:43.586288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40935 (local address 127.0.0.1:34497)
0620 03:10:43.586462 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:43.586469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:43.586819 (+ 350us) server_negotiation.cc:407] Connection header received
0620 03:10:43.587017 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:43.587026 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:43.587078 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:43.587178 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:43.588139 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:43.588146 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:43.588154 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:43.588168 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:43.588204 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:43.588218 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:43.588333 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:43.588449 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 03:10:43.588506 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 03:10:43.588801 21069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:43.590184 21069 log.cc:824] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:43.592727 21069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4: No bootstrap required, opened a new log
I20260620 03:10:43.600484 21069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164535852e7a41208730603159995ed4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } }
I20260620 03:10:43.600960 21069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:43.601078 21069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 164535852e7a41208730603159995ed4, State: Initialized, Role: FOLLOWER
I20260620 03:10:43.601671 21069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164535852e7a41208730603159995ed4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } }
I20260620 03:10:43.601898 21069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:10:43.602046 21069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:10:43.602227 21069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:43.604456 21069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164535852e7a41208730603159995ed4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } }
I20260620 03:10:43.604990 21069 leader_election.cc:302] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 164535852e7a41208730603159995ed4; no voters:
I20260620 03:10:43.605692 21069 leader_election.cc:288] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:10:43.605780 21079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:43.606487 21079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4 [term 1 LEADER]: Becoming Leader. State: Replica: 164535852e7a41208730603159995ed4, State: Running, Role: LEADER
I20260620 03:10:43.607025 21079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164535852e7a41208730603159995ed4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } }
I20260620 03:10:43.607789 21069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:10:43.609887 21081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 164535852e7a41208730603159995ed4. Latest consensus state: current_term: 1 leader_uuid: "164535852e7a41208730603159995ed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164535852e7a41208730603159995ed4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } } }
I20260620 03:10:43.610368 21081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:43.609987 21080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "164535852e7a41208730603159995ed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164535852e7a41208730603159995ed4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } } }
I20260620 03:10:43.611488 21080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:43.613106 21087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:10:43.618070 21087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:10:43.626969 21087 catalog_manager.cc:1347] Generated new cluster ID: e6edb5d6fd4a4861b44491e63fafdad5
I20260620 03:10:43.627122 21087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:10:43.651037 21087 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:10:43.651544 21087 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:10:43.657473 21087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 164535852e7a41208730603159995ed4: Generated new TSK 0
I20260620 03:10:43.658214 21087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:10:43.662490 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 21002
W20260620 03:10:43.663558 21064 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56649
W20260620 03:10:43.664258 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 21002: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:10:43.664409 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 21002
2026-06-20T03:10:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1845 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:10:43 dist-test-slave-xv68 krb5kdc[21100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:10:43 dist-test-slave-xv68 krb5kdc[21100](info): set up 2 sockets
Jun 20 03:10:43 dist-test-slave-xv68 krb5kdc[21100](info): commencing operation
krb5kdc: starting...
W20260620 03:10:44.759685 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:10:44 dist-test-slave-xv68 krb5kdc[21100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:10:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:10:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:10:45.006661 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45647
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:45.330374 21113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:45.330637 21113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:45.330797 21113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:45.337792 21113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:10:45.337898 21113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:45.338198 21113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:45.338325 21113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:10:45.338418 21113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:10:45.348089 21113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:45.349411 21113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:45.350696 21113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:45.362282 21118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:45.362293 21121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:45.362282 21119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:45.363140 21113 server_base.cc:1034] running on GCE node
I20260620 03:10:45.363883 21113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:45.365199 21113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:45.366428 21113 hybrid_clock.cc:648] HybridClock initialized: now 1781925045366379 us; error 72 us; skew 500 ppm
I20260620 03:10:45.366775 21113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:45 dist-test-slave-xv68 krb5kdc[21100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:45.370143 21113 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:10:45.371654 21113 webserver.cc:460] Webserver started at http://127.0.0.1:35399/ using document root <none> and password file <none>
I20260620 03:10:45.372309 21113 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:45.372457 21113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:45.372766 21113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:45.375667 21113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "4c8d2ea1f02f438fa0962dd178d021ce"
format_stamp: "Formatted at 2026-06-20 03:10:45 on dist-test-slave-xv68"
server_key: "c905a2b462935c5d29ce452e6feb39ab"
server_key_iv: "85eb7fc48dae0dfc06bf1d3d570b519b"
server_key_version: "encryptionkey@0"
I20260620 03:10:45.376403 21113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "4c8d2ea1f02f438fa0962dd178d021ce"
format_stamp: "Formatted at 2026-06-20 03:10:45 on dist-test-slave-xv68"
server_key: "c905a2b462935c5d29ce452e6feb39ab"
server_key_iv: "85eb7fc48dae0dfc06bf1d3d570b519b"
server_key_version: "encryptionkey@0"
I20260620 03:10:45.381358 21113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:10:45.384603 21128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:45.385811 21113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 03:10:45.385991 21113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "4c8d2ea1f02f438fa0962dd178d021ce"
format_stamp: "Formatted at 2026-06-20 03:10:45 on dist-test-slave-xv68"
server_key: "c905a2b462935c5d29ce452e6feb39ab"
server_key_iv: "85eb7fc48dae0dfc06bf1d3d570b519b"
server_key_version: "encryptionkey@0"
I20260620 03:10:45.386178 21113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:45.405035 21113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:45.409353 21113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:45.409638 21113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:45.427073 21113 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45647
I20260620 03:10:45.427098 21179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45647 every 8 connection(s)
I20260620 03:10:45.428481 21113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:10:45.429924 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 21113
I20260620 03:10:45.430119 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:10:45.430528 388 external_mini_cluster.cc:1428] Setting key e32f889e48b9767703e46f0445c11381
I20260620 03:10:45.433002 21180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 03:10:45 dist-test-slave-xv68 krb5kdc[21100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 03:10:45.445662 21180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce: Bootstrap starting.
I20260620 03:10:45.449721 21180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:45.451036 21180 log.cc:824] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:45.454526 21180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce: No bootstrap required, opened a new log
I20260620 03:10:45.455497 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:45.432221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55421 (local address 127.0.0.1:45647)
0620 03:10:45.433024 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:45.433041 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:45.433092 (+ 51us) server_negotiation.cc:407] Connection header received
0620 03:10:45.435447 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:45.435481 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:45.436016 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:45.436449 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:45.437782 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:45.439837 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:45.441132 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:45.442131 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:45.442322 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:45.446421 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:45.446431 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:45.446449 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:45.446508 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:45.450639 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:45.451533 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:45.451543 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:45.451552 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:45.451812 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:45.452307 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:45.452310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:45.452312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:45.452996 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:45.453282 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:45.453692 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 03:10:45.454115 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":338,"threads_started":1}
I20260620 03:10:45.459178 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:10:45.463202 21180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c8d2ea1f02f438fa0962dd178d021ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45647 } }
I20260620 03:10:45.463701 21180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:45.463827 21180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8d2ea1f02f438fa0962dd178d021ce, State: Initialized, Role: FOLLOWER
I20260620 03:10:45.464439 21180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c8d2ea1f02f438fa0962dd178d021ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45647 } }
I20260620 03:10:45.464722 21180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:10:45.464879 21180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:10:45.465062 21180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:45.467777 21180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c8d2ea1f02f438fa0962dd178d021ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45647 } }
I20260620 03:10:45.468271 21180 leader_election.cc:302] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c8d2ea1f02f438fa0962dd178d021ce; no voters:
I20260620 03:10:45.468845 21180 leader_election.cc:288] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:10:45.469005 21190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:45.469689 21190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce [term 1 LEADER]: Becoming Leader. State: Replica: 4c8d2ea1f02f438fa0962dd178d021ce, State: Running, Role: LEADER
I20260620 03:10:45.470857 21190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c8d2ea1f02f438fa0962dd178d021ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45647 } }
I20260620 03:10:45.471691 21180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:10:45.473841 21191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c8d2ea1f02f438fa0962dd178d021ce. Latest consensus state: current_term: 1 leader_uuid: "4c8d2ea1f02f438fa0962dd178d021ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c8d2ea1f02f438fa0962dd178d021ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45647 } } }
I20260620 03:10:45.474220 21191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:45.477001 21192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c8d2ea1f02f438fa0962dd178d021ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c8d2ea1f02f438fa0962dd178d021ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45647 } } }
I20260620 03:10:45.477451 21192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:45.479027 21195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:10:45.481597 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:45.463769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39095 (local address 127.0.0.1:45647)
0620 03:10:45.464010 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:45.464019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:45.464526 (+ 507us) server_negotiation.cc:407] Connection header received
0620 03:10:45.464630 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:45.464636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:45.464723 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:45.464836 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:45.466335 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:45.467484 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:45.469119 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:45.470166 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:45.470347 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:45.473537 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:45.473546 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:45.473553 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:45.473600 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:45.478435 (+ 4835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:45.479232 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:45.479239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:45.479243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:45.479448 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:45.479989 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:45.479994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:45.479996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:45.480374 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:45.480671 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:45.480973 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 03:10:45.481191 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 03:10:45.486093 21195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:10:45.496819 21195 catalog_manager.cc:1347] Generated new cluster ID: 6c6afcc24e1d4c98b69d940b40498ffa
I20260620 03:10:45.496940 21195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:10:45.511653 21195 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:10:45.513485 21195 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:10:45.521631 21195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c8d2ea1f02f438fa0962dd178d021ce: Generated new TSK 0
I20260620 03:10:45.522413 21195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:10:45.539600 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 21113
I20260620 03:10:45.541011 21175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:45.632462 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 21113
2026-06-20T03:10:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1971 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:10:45 dist-test-slave-xv68 krb5kdc[21212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:10:45 dist-test-slave-xv68 krb5kdc[21212](info): set up 2 sockets
Jun 20 03:10:45 dist-test-slave-xv68 krb5kdc[21212](info): commencing operation
krb5kdc: starting...
W20260620 03:10:46.739630 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:10:46 dist-test-slave-xv68 krb5kdc[21212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:10:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:10:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:10:46.978566 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37675
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:47.299753 21225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:47.300104 21225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:47.300282 21225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:47.307363 21225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:10:47.307524 21225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:47.307633 21225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:47.307744 21225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:10:47.307832 21225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:10:47.317529 21225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:47.318815 21225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:47.320128 21225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:47.331880 21231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:47.331972 21233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:47.331995 21230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:47.332729 21225 server_base.cc:1034] running on GCE node
I20260620 03:10:47.333449 21225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:47.334797 21225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:47.336016 21225 hybrid_clock.cc:648] HybridClock initialized: now 1781925047335965 us; error 51 us; skew 500 ppm
I20260620 03:10:47.336498 21225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:47 dist-test-slave-xv68 krb5kdc[21212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:47.339671 21225 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:10:47.341075 21225 webserver.cc:460] Webserver started at http://127.0.0.1:44375/ using document root <none> and password file <none>
I20260620 03:10:47.341732 21225 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:47.341828 21225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:47.342070 21225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:47.345037 21225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "ec20a9cd520148918820b982436602c5"
format_stamp: "Formatted at 2026-06-20 03:10:47 on dist-test-slave-xv68"
server_key: "9fbda50383d045d39557ead55cfb296a"
server_key_iv: "71cc5456f03b04afdadeed8bc854094c"
server_key_version: "encryptionkey@0"
I20260620 03:10:47.345752 21225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "ec20a9cd520148918820b982436602c5"
format_stamp: "Formatted at 2026-06-20 03:10:47 on dist-test-slave-xv68"
server_key: "9fbda50383d045d39557ead55cfb296a"
server_key_iv: "71cc5456f03b04afdadeed8bc854094c"
server_key_version: "encryptionkey@0"
I20260620 03:10:47.350869 21225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 03:10:47.354380 21240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:47.355827 21225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:10:47.356000 21225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "ec20a9cd520148918820b982436602c5"
format_stamp: "Formatted at 2026-06-20 03:10:47 on dist-test-slave-xv68"
server_key: "9fbda50383d045d39557ead55cfb296a"
server_key_iv: "71cc5456f03b04afdadeed8bc854094c"
server_key_version: "encryptionkey@0"
I20260620 03:10:47.356166 21225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:47.368642 21225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:47.372448 21225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:47.372668 21225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:47.391031 21225 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37675
I20260620 03:10:47.391078 21291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37675 every 8 connection(s)
I20260620 03:10:47.392429 21225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:10:47.397105 21292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:47.400421 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 21225
I20260620 03:10:47.400614 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:10:47.401043 388 external_mini_cluster.cc:1428] Setting key b5978f29a9fa6ff9bf7dc0ff76d10340
I20260620 03:10:47.412694 21292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5: Bootstrap starting.
I20260620 03:10:47.414120 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:47.402792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55213 (local address 127.0.0.1:37675)
0620 03:10:47.403656 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:47.403667 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:47.403713 (+ 46us) server_negotiation.cc:407] Connection header received
0620 03:10:47.405296 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:47.405316 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:47.405701 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:47.406009 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:47.406598 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:47.407897 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:47.409856 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:47.410696 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:47.410816 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:47.411976 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:47.411984 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:47.411995 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:47.412510 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:47.412582 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:47.412614 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:47.412701 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:47.412956 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 03:10:47.413230 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":414,"threads_started":1}
I20260620 03:10:47.415851 21292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:47.416992 21292 log.cc:824] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:47.417790 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:10:47.420030 21292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5: No bootstrap required, opened a new log
I20260620 03:10:47.429400 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:47.422895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40305 (local address 127.0.0.1:37675)
0620 03:10:47.423097 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:47.423103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:47.423478 (+ 375us) server_negotiation.cc:407] Connection header received
0620 03:10:47.423599 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:47.423604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:47.423674 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:47.423779 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:47.424322 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:47.425400 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:47.426796 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:47.427516 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:47.427656 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:47.428695 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:47.428702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:47.428706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:47.428726 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:47.428771 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:47.428790 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:47.428876 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:47.428990 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 03:10:47.429080 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 03:10:47.429631 21292 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec20a9cd520148918820b982436602c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37675 } }
I20260620 03:10:47.430051 21292 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:47.430171 21292 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec20a9cd520148918820b982436602c5, State: Initialized, Role: FOLLOWER
I20260620 03:10:47.430738 21292 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec20a9cd520148918820b982436602c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37675 } }
I20260620 03:10:47.430974 21292 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:10:47.431126 21292 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:10:47.431324 21292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:47.433444 21292 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec20a9cd520148918820b982436602c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37675 } }
I20260620 03:10:47.433954 21292 leader_election.cc:302] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec20a9cd520148918820b982436602c5; no voters:
I20260620 03:10:47.434592 21292 leader_election.cc:288] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:10:47.434706 21302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:47.435684 21302 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5 [term 1 LEADER]: Becoming Leader. State: Replica: ec20a9cd520148918820b982436602c5, State: Running, Role: LEADER
I20260620 03:10:47.436278 21302 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec20a9cd520148918820b982436602c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37675 } }
I20260620 03:10:47.436802 21292 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:10:47.439043 21304 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec20a9cd520148918820b982436602c5. Latest consensus state: current_term: 1 leader_uuid: "ec20a9cd520148918820b982436602c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec20a9cd520148918820b982436602c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37675 } } }
I20260620 03:10:47.439945 21304 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:47.439531 21303 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec20a9cd520148918820b982436602c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec20a9cd520148918820b982436602c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37675 } } }
I20260620 03:10:47.440420 21303 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:47.440892 21311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:10:47.447484 21311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:10:47.456929 21311 catalog_manager.cc:1347] Generated new cluster ID: 07b9fc5f36624db59c8ee70e8e06ab63
I20260620 03:10:47.457057 21311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:10:47.468495 21311 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:10:47.469802 21311 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:10:47.475114 21311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec20a9cd520148918820b982436602c5: Generated new TSK 0
I20260620 03:10:47.475891 21311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:10:47.488073 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 21225
W20260620 03:10:47.489586 21287 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55213
W20260620 03:10:47.490326 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 21225: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:10:47.490445 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 21225
2026-06-20T03:10:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1858 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:10:47 dist-test-slave-xv68 krb5kdc[21323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:10:47 dist-test-slave-xv68 krb5kdc[21323](info): set up 2 sockets
Jun 20 03:10:47 dist-test-slave-xv68 krb5kdc[21323](info): commencing operation
krb5kdc: starting...
W20260620 03:10:48.599650 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:10:48 dist-test-slave-xv68 krb5kdc[21323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:10:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:10:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:10:48.834429 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:49.154856 21336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:49.155187 21336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:49.155378 21336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:49.162446 21336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:10:49.162550 21336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:49.162634 21336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:49.162743 21336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:10:49.162835 21336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:10:49.172446 21336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:49.173702 21336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:49.175002 21336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:49.186803 21341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:49.186753 21342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:49.186832 21344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:49.187992 21336 server_base.cc:1034] running on GCE node
I20260620 03:10:49.188724 21336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:49.190042 21336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:49.191272 21336 hybrid_clock.cc:648] HybridClock initialized: now 1781925049191232 us; error 50 us; skew 500 ppm
I20260620 03:10:49.191665 21336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:49 dist-test-slave-xv68 krb5kdc[21323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:49.199553 21336 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:10:49.201073 21336 webserver.cc:460] Webserver started at http://127.0.0.1:39533/ using document root <none> and password file <none>
I20260620 03:10:49.201741 21336 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:49.201886 21336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:49.202195 21336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:49.205133 21336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "41df5299ed2f433c9ab320e4530445be"
format_stamp: "Formatted at 2026-06-20 03:10:49 on dist-test-slave-xv68"
server_key: "a355efd18cf773ef2f83cc94774ce80f"
server_key_iv: "d7b5777469c830427a9c09487cf64ce8"
server_key_version: "encryptionkey@0"
I20260620 03:10:49.205951 21336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "41df5299ed2f433c9ab320e4530445be"
format_stamp: "Formatted at 2026-06-20 03:10:49 on dist-test-slave-xv68"
server_key: "a355efd18cf773ef2f83cc94774ce80f"
server_key_iv: "d7b5777469c830427a9c09487cf64ce8"
server_key_version: "encryptionkey@0"
I20260620 03:10:49.211081 21336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 03:10:49.214542 21351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:49.215830 21336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:10:49.216027 21336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "41df5299ed2f433c9ab320e4530445be"
format_stamp: "Formatted at 2026-06-20 03:10:49 on dist-test-slave-xv68"
server_key: "a355efd18cf773ef2f83cc94774ce80f"
server_key_iv: "d7b5777469c830427a9c09487cf64ce8"
server_key_version: "encryptionkey@0"
I20260620 03:10:49.216193 21336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:49.221728 21336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:49.225505 21336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:49.225785 21336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:49.244396 21336 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42259
I20260620 03:10:49.244442 21402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42259 every 8 connection(s)
I20260620 03:10:49.245749 21336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:10:49.246737 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 21336
I20260620 03:10:49.246915 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:10:49.247442 388 external_mini_cluster.cc:1428] Setting key 897fc5fba6dd59c505a9e6be5d66c225
I20260620 03:10:49.250200 21403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:49.256121 21406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:49.249229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38237 (local address 127.0.0.1:42259)
0620 03:10:49.250054 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:49.250068 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:49.250119 (+ 51us) server_negotiation.cc:407] Connection header received
0620 03:10:49.252399 (+ 2280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:49.252424 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:49.252822 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:49.253260 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:49.254201 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:49.254209 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:49.254226 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:49.254243 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:49.254331 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:49.254367 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:49.254591 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:49.254733 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 03:10:49.254905 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":346,"threads_started":1}
I20260620 03:10:49.259573 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:10:49.262780 21403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be: Bootstrap starting.
I20260620 03:10:49.266258 21406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:49.263669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32833 (local address 127.0.0.1:42259)
0620 03:10:49.263864 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:49.263871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:49.264370 (+ 499us) server_negotiation.cc:407] Connection header received
0620 03:10:49.264538 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:49.264542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:49.264583 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:49.264671 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:49.265574 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:49.265578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:49.265581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:49.265596 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:49.265631 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:49.265646 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:49.265746 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:49.265902 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 03:10:49.265948 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 03:10:49.266571 21403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:49.267789 21403 log.cc:824] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:49.270284 21403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be: No bootstrap required, opened a new log
I20260620 03:10:49.277292 21403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41df5299ed2f433c9ab320e4530445be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42259 } }
I20260620 03:10:49.277611 21403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:49.277683 21403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41df5299ed2f433c9ab320e4530445be, State: Initialized, Role: FOLLOWER
I20260620 03:10:49.278172 21403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41df5299ed2f433c9ab320e4530445be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42259 } }
I20260620 03:10:49.278414 21403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:10:49.278579 21403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:10:49.278774 21403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:49.280941 21403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41df5299ed2f433c9ab320e4530445be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42259 } }
I20260620 03:10:49.281404 21403 leader_election.cc:302] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41df5299ed2f433c9ab320e4530445be; no voters:
I20260620 03:10:49.281946 21403 leader_election.cc:288] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:10:49.282101 21413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:49.282788 21413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be [term 1 LEADER]: Becoming Leader. State: Replica: 41df5299ed2f433c9ab320e4530445be, State: Running, Role: LEADER
I20260620 03:10:49.283358 21413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41df5299ed2f433c9ab320e4530445be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42259 } }
I20260620 03:10:49.283936 21403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:10:49.287786 21415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41df5299ed2f433c9ab320e4530445be. Latest consensus state: current_term: 1 leader_uuid: "41df5299ed2f433c9ab320e4530445be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41df5299ed2f433c9ab320e4530445be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42259 } } }
I20260620 03:10:49.288015 21414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41df5299ed2f433c9ab320e4530445be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41df5299ed2f433c9ab320e4530445be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42259 } } }
I20260620 03:10:49.288331 21415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:49.288380 21414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:49.289389 21422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:10:49.294257 21422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:10:49.303154 21422 catalog_manager.cc:1347] Generated new cluster ID: 8e5d03c23bc2438da5135b86790dcd8d
I20260620 03:10:49.303284 21422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:10:49.358448 21422 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:10:49.358978 21422 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:10:49.367748 21422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41df5299ed2f433c9ab320e4530445be: Generated new TSK 0
I20260620 03:10:49.368561 21422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:10:49.377868 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 21336
W20260620 03:10:49.379106 21398 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38237
W20260620 03:10:49.379798 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 21336: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:10:49.379944 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 21336
2026-06-20T03:10:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1890 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:10:49 dist-test-slave-xv68 krb5kdc[21434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:10:49 dist-test-slave-xv68 krb5kdc[21434](info): set up 2 sockets
Jun 20 03:10:49 dist-test-slave-xv68 krb5kdc[21434](info): commencing operation
krb5kdc: starting...
W20260620 03:10:50.483688 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:10:50 dist-test-slave-xv68 krb5kdc[21434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:10:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:10:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:10:50.734472 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44191
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:51.058758 21447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:51.059023 21447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:51.059196 21447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:51.066147 21447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:10:51.066241 21447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:51.066360 21447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:51.066470 21447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:10:51.066529 21447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:10:51.075881 21447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:51.077118 21447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:51.078421 21447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:51.089542 21452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:51.089561 21453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:51.089526 21455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:51.090669 21447 server_base.cc:1034] running on GCE node
I20260620 03:10:51.091409 21447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:51.092648 21447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:51.093932 21447 hybrid_clock.cc:648] HybridClock initialized: now 1781925051093877 us; error 48 us; skew 500 ppm
I20260620 03:10:51.094321 21447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:51 dist-test-slave-xv68 krb5kdc[21434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:51.097453 21447 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:10:51.098973 21447 webserver.cc:460] Webserver started at http://127.0.0.1:38037/ using document root <none> and password file <none>
I20260620 03:10:51.099707 21447 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:51.099856 21447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:51.100178 21447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:51.103118 21447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "f78709cba4004f85829a8b0c63297d2e"
format_stamp: "Formatted at 2026-06-20 03:10:51 on dist-test-slave-xv68"
server_key: "13363988f5f6d9928eac446fd0f40d0e"
server_key_iv: "80abcdf14715131a7a1d5f374c1ad6fb"
server_key_version: "encryptionkey@0"
I20260620 03:10:51.103940 21447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "f78709cba4004f85829a8b0c63297d2e"
format_stamp: "Formatted at 2026-06-20 03:10:51 on dist-test-slave-xv68"
server_key: "13363988f5f6d9928eac446fd0f40d0e"
server_key_iv: "80abcdf14715131a7a1d5f374c1ad6fb"
server_key_version: "encryptionkey@0"
I20260620 03:10:51.108805 21447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 03:10:51.112031 21462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:51.113224 21447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 03:10:51.113406 21447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "f78709cba4004f85829a8b0c63297d2e"
format_stamp: "Formatted at 2026-06-20 03:10:51 on dist-test-slave-xv68"
server_key: "13363988f5f6d9928eac446fd0f40d0e"
server_key_iv: "80abcdf14715131a7a1d5f374c1ad6fb"
server_key_version: "encryptionkey@0"
I20260620 03:10:51.113591 21447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:51.134456 21447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:51.138898 21447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:51.139199 21447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:51.157714 21447 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44191
I20260620 03:10:51.157758 21513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44191 every 8 connection(s)
I20260620 03:10:51.159039 21447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:10:51.162900 21514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:51.167476 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 21447
I20260620 03:10:51.167670 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:10:51.168087 388 external_mini_cluster.cc:1428] Setting key 391c13a2dfdcf3b8a4866e45fade2724
I20260620 03:10:51.174943 21514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e: Bootstrap starting.
I20260620 03:10:51.178608 21514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:51.179796 21514 log.cc:824] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:51.182703 21514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e: No bootstrap required, opened a new log
Jun 20 03:10:51 dist-test-slave-xv68 krb5kdc[21434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 03:10:51.192353 21514 raft_consensus.cc:348] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78709cba4004f85829a8b0c63297d2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44191 } }
I20260620 03:10:51.192720 21514 raft_consensus.cc:374] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:51.192857 21514 raft_consensus.cc:729] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78709cba4004f85829a8b0c63297d2e, State: Initialized, Role: FOLLOWER
I20260620 03:10:51.192900 21517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:51.169949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:59859 (local address 127.0.0.1:44191)
0620 03:10:51.170723 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:51.170738 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:51.170791 (+ 53us) server_negotiation.cc:407] Connection header received
0620 03:10:51.173366 (+ 2575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:51.173393 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:51.173892 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:51.174307 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:51.175740 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:51.177760 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:51.179178 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:51.180236 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:51.180459 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:51.185063 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:51.185073 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:51.185088 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:51.185138 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:51.188370 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:51.189327 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:51.189333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:51.189340 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:51.189513 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:51.189976 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:51.189980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:51.189982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:51.190518 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:51.190755 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:51.191439 (+ 684us) server_negotiation.cc:299] Negotiation successful
0620 03:10:51.191796 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":318,"threads_started":1}
I20260620 03:10:51.193495 21514 consensus_queue.cc:260] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78709cba4004f85829a8b0c63297d2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44191 } }
I20260620 03:10:51.193763 21514 raft_consensus.cc:388] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:10:51.193912 21514 raft_consensus.cc:482] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:10:51.194092 21514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:51.196762 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:10:51.196756 21514 raft_consensus.cc:504] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78709cba4004f85829a8b0c63297d2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44191 } }
I20260620 03:10:51.197324 21514 leader_election.cc:302] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f78709cba4004f85829a8b0c63297d2e; no voters:
I20260620 03:10:51.197896 21514 leader_election.cc:288] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:10:51.198043 21519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:51.199911 21514 sys_catalog.cc:564] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:10:51.203533 21519 raft_consensus.cc:686] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e [term 1 LEADER]: Becoming Leader. State: Replica: f78709cba4004f85829a8b0c63297d2e, State: Running, Role: LEADER
I20260620 03:10:51.204372 21519 consensus_queue.cc:237] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78709cba4004f85829a8b0c63297d2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44191 } }
I20260620 03:10:51.212776 21521 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f78709cba4004f85829a8b0c63297d2e. Latest consensus state: current_term: 1 leader_uuid: "f78709cba4004f85829a8b0c63297d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78709cba4004f85829a8b0c63297d2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44191 } } }
I20260620 03:10:51.213220 21521 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:51.213809 21520 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f78709cba4004f85829a8b0c63297d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78709cba4004f85829a8b0c63297d2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44191 } } }
I20260620 03:10:51.214118 21520 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:51.225363 21538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:10:51.232053 21538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:10:51.233140 21517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:51.206046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:47127 (local address 127.0.0.1:44191)
0620 03:10:51.210063 (+ 4017us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:51.210073 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:51.211067 (+ 994us) server_negotiation.cc:407] Connection header received
0620 03:10:51.211407 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:51.211417 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:51.211554 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:51.211675 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:51.213815 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:51.215073 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:51.217542 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:51.218395 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:51.218558 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:51.222516 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:51.222526 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:51.222534 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:51.222587 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:51.227559 (+ 4972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:51.228880 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:51.228889 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:51.228895 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:51.229100 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:51.229817 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:51.229824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:51.229830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:51.230395 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:51.232124 (+ 1729us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:51.232349 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 03:10:51.232587 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3848}
I20260620 03:10:51.241428 21538 catalog_manager.cc:1347] Generated new cluster ID: 3c82da49edc441e08fc3ed076c8d16a1
I20260620 03:10:51.241729 21538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:10:51.251814 21538 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:10:51.253614 21538 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:10:51.261250 21538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f78709cba4004f85829a8b0c63297d2e: Generated new TSK 0
I20260620 03:10:51.262080 21538 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:10:51.274204 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 21447
I20260620 03:10:51.275673 21509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:51.364704 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 21447
W20260620 03:10:51.383103 21432 connection.cc:535] client connection to 127.0.0.1:44191 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44191): Connection reset by peer (error 104)
2026-06-20T03:10:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1984 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:10:51 dist-test-slave-xv68 krb5kdc[21547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:10:51 dist-test-slave-xv68 krb5kdc[21547](info): set up 2 sockets
Jun 20 03:10:51 dist-test-slave-xv68 krb5kdc[21547](info): commencing operation
krb5kdc: starting...
W20260620 03:10:52.467685 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:10:52 dist-test-slave-xv68 krb5kdc[21547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:10:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:10:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:10:52.710666 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34197
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:53.030550 21560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:53.030857 21560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:53.031023 21560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:53.038295 21560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:10:53.038453 21560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:53.038564 21560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:53.038671 21560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:10:53.038754 21560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:10:53.048585 21560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:53.049844 21560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:53.051151 21560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:53.063308 21568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:53.063371 21565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:53.063395 21566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:53.064621 21560 server_base.cc:1034] running on GCE node
I20260620 03:10:53.065423 21560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:53.066746 21560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:53.068020 21560 hybrid_clock.cc:648] HybridClock initialized: now 1781925053067982 us; error 50 us; skew 500 ppm
I20260620 03:10:53.068430 21560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:53 dist-test-slave-xv68 krb5kdc[21547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:53.071481 21560 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:10:53.072927 21560 webserver.cc:460] Webserver started at http://127.0.0.1:44225/ using document root <none> and password file <none>
I20260620 03:10:53.073647 21560 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:53.073787 21560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:53.074083 21560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:53.076956 21560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "201aea89053b41ab9884f182e60f4155"
format_stamp: "Formatted at 2026-06-20 03:10:53 on dist-test-slave-xv68"
server_key: "0198df48e19df68e8def1f75044571e8"
server_key_iv: "9b71634ae7bcb2720dc8580d9c7f8b91"
server_key_version: "encryptionkey@0"
I20260620 03:10:53.077687 21560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "201aea89053b41ab9884f182e60f4155"
format_stamp: "Formatted at 2026-06-20 03:10:53 on dist-test-slave-xv68"
server_key: "0198df48e19df68e8def1f75044571e8"
server_key_iv: "9b71634ae7bcb2720dc8580d9c7f8b91"
server_key_version: "encryptionkey@0"
I20260620 03:10:53.082614 21560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 03:10:53.085985 21575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:53.087201 21560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:10:53.087414 21560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "201aea89053b41ab9884f182e60f4155"
format_stamp: "Formatted at 2026-06-20 03:10:53 on dist-test-slave-xv68"
server_key: "0198df48e19df68e8def1f75044571e8"
server_key_iv: "9b71634ae7bcb2720dc8580d9c7f8b91"
server_key_version: "encryptionkey@0"
I20260620 03:10:53.087602 21560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:53.109670 21560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:53.113633 21560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:53.113965 21560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:53.132161 21560 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34197
I20260620 03:10:53.132191 21626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34197 every 8 connection(s)
I20260620 03:10:53.133466 21560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:10:53.134526 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 21560
I20260620 03:10:53.134725 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:10:53.135246 388 external_mini_cluster.cc:1428] Setting key 2bb2f562cbb7dca4a7c5355f2e6f5bc2
I20260620 03:10:53.138005 21627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:53.148062 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:53.136964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:41495 (local address 127.0.0.1:34197)
0620 03:10:53.137717 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:53.137728 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:53.137767 (+ 39us) server_negotiation.cc:407] Connection header received
0620 03:10:53.139592 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:53.139613 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:53.139958 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:53.140355 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:53.140886 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:53.142177 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:53.143584 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:53.144211 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:53.144399 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:53.145734 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:53.145742 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:53.145754 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:53.146300 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:53.146374 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:53.146398 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:53.146702 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:53.146961 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 03:10:53.147145 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":332,"threads_started":1}
I20260620 03:10:53.148516 21627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155: Bootstrap starting.
I20260620 03:10:53.151772 21627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:53.151815 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:10:53.152727 21627 log.cc:824] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:53.155233 21627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155: No bootstrap required, opened a new log
I20260620 03:10:53.163204 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:53.156268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:42159 (local address 127.0.0.1:34197)
0620 03:10:53.156478 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:53.156484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:53.156837 (+ 353us) server_negotiation.cc:407] Connection header received
0620 03:10:53.157007 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:53.157011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:53.157070 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:53.157167 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:53.157792 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:53.158635 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:53.160702 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:53.161177 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:53.161277 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:53.162423 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:53.162427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:53.162431 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:53.162460 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:53.162508 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:53.162522 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:53.162603 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:53.162806 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 03:10:53.162849 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 03:10:53.164891 21627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201aea89053b41ab9884f182e60f4155" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34197 } }
I20260620 03:10:53.165263 21627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:53.165373 21627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201aea89053b41ab9884f182e60f4155, State: Initialized, Role: FOLLOWER
I20260620 03:10:53.165900 21627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201aea89053b41ab9884f182e60f4155" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34197 } }
I20260620 03:10:53.166138 21627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:10:53.166299 21627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:10:53.166482 21627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:53.169020 21627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201aea89053b41ab9884f182e60f4155" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34197 } }
I20260620 03:10:53.169495 21627 leader_election.cc:302] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 201aea89053b41ab9884f182e60f4155; no voters:
I20260620 03:10:53.170040 21627 leader_election.cc:288] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:10:53.170171 21637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:53.170861 21637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155 [term 1 LEADER]: Becoming Leader. State: Replica: 201aea89053b41ab9884f182e60f4155, State: Running, Role: LEADER
I20260620 03:10:53.171533 21637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201aea89053b41ab9884f182e60f4155" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34197 } }
I20260620 03:10:53.172303 21627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:10:53.174594 21639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 201aea89053b41ab9884f182e60f4155. Latest consensus state: current_term: 1 leader_uuid: "201aea89053b41ab9884f182e60f4155" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201aea89053b41ab9884f182e60f4155" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34197 } } }
I20260620 03:10:53.174557 21638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "201aea89053b41ab9884f182e60f4155" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201aea89053b41ab9884f182e60f4155" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34197 } } }
I20260620 03:10:53.174993 21639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:53.175035 21638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:53.176580 21645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:10:53.181116 21645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:10:53.189514 21645 catalog_manager.cc:1347] Generated new cluster ID: 90bfe9d8638e4bdcb63a87f4c543b80a
I20260620 03:10:53.189639 21645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:10:53.214432 21645 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:10:53.216260 21645 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:10:53.225277 21645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 201aea89053b41ab9884f182e60f4155: Generated new TSK 0
I20260620 03:10:53.226253 21645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:10:53.235770 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 21560
W20260620 03:10:53.236901 21622 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.102:41495
W20260620 03:10:53.237564 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 21560: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:10:53.237709 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 21560
W20260620 03:10:53.254455 21545 connection.cc:535] client connection to 127.0.0.1:34197 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34197): Connection reset by peer (error 104)
2026-06-20T03:10:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1870 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:10:53 dist-test-slave-xv68 krb5kdc[21658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:10:53 dist-test-slave-xv68 krb5kdc[21658](info): set up 2 sockets
Jun 20 03:10:53 dist-test-slave-xv68 krb5kdc[21658](info): commencing operation
krb5kdc: starting...
W20260620 03:10:54.339718 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:10:54 dist-test-slave-xv68 krb5kdc[21658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:10:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:10:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:10:54.586179 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46403
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:54.906544 21671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:54.906883 21671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:54.907056 21671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:54.914320 21671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:10:54.914479 21671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:54.914587 21671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:54.914698 21671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:10:54.914788 21671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:10:54.924755 21671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:54.926052 21671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:54.927393 21671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:54.939628 21677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:54.939703 21676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:54.939680 21679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:54.940392 21671 server_base.cc:1034] running on GCE node
I20260620 03:10:54.941088 21671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:54.942401 21671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:54.943593 21671 hybrid_clock.cc:648] HybridClock initialized: now 1781925054943556 us; error 33 us; skew 500 ppm
I20260620 03:10:54.943897 21671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:54 dist-test-slave-xv68 krb5kdc[21658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:54.950510 21671 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:10:54.952157 21671 webserver.cc:460] Webserver started at http://127.0.0.1:35211/ using document root <none> and password file <none>
I20260620 03:10:54.952850 21671 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:54.953001 21671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:54.953320 21671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:54.956319 21671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "47a89ef086b54fe996c64e4a958d8620"
format_stamp: "Formatted at 2026-06-20 03:10:54 on dist-test-slave-xv68"
server_key: "d0178a57b3fd471f8616992cee8f2db0"
server_key_iv: "ccaad6409ec2764da6895fe301c54ba4"
server_key_version: "encryptionkey@0"
I20260620 03:10:54.957060 21671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "47a89ef086b54fe996c64e4a958d8620"
format_stamp: "Formatted at 2026-06-20 03:10:54 on dist-test-slave-xv68"
server_key: "d0178a57b3fd471f8616992cee8f2db0"
server_key_iv: "ccaad6409ec2764da6895fe301c54ba4"
server_key_version: "encryptionkey@0"
I20260620 03:10:54.962067 21671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:10:54.965358 21686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:54.966544 21671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 03:10:54.966749 21671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "47a89ef086b54fe996c64e4a958d8620"
format_stamp: "Formatted at 2026-06-20 03:10:54 on dist-test-slave-xv68"
server_key: "d0178a57b3fd471f8616992cee8f2db0"
server_key_iv: "ccaad6409ec2764da6895fe301c54ba4"
server_key_version: "encryptionkey@0"
I20260620 03:10:54.966943 21671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:54.972574 21671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:54.976667 21671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:54.976965 21671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:54.996163 21671 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46403
I20260620 03:10:54.996208 21737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46403 every 8 connection(s)
I20260620 03:10:54.997524 21671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:10:54.998481 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 21671
I20260620 03:10:54.998675 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:10:54.999203 388 external_mini_cluster.cc:1428] Setting key fa3da07d99d76d35ac3cb306c4a5079a
I20260620 03:10:55.001947 21738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:55.006476 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:55.000929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:59767 (local address 127.0.0.1:46403)
0620 03:10:55.001690 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:55.001702 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:55.001740 (+ 38us) server_negotiation.cc:407] Connection header received
0620 03:10:55.003413 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:55.003434 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:55.003731 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:55.004018 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:55.005014 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:55.005020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:55.005035 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:55.005048 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:55.005121 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:55.005142 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:55.005307 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:55.005410 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 03:10:55.005605 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":349,"threads_started":1}
I20260620 03:10:55.009924 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:10:55.014410 21738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620: Bootstrap starting.
I20260620 03:10:55.017596 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:55.015226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:49061 (local address 127.0.0.1:46403)
0620 03:10:55.015455 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:55.015461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:55.015740 (+ 279us) server_negotiation.cc:407] Connection header received
0620 03:10:55.015898 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:55.015902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:55.015949 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:55.016028 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:55.016949 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:55.016953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:55.016956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:55.016970 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:55.017004 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:55.017018 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:55.017102 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:55.017243 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 03:10:55.017293 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260620 03:10:55.018281 21738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:55.019366 21738 log.cc:824] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:55.021816 21738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620: No bootstrap required, opened a new log
I20260620 03:10:55.028991 21738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a89ef086b54fe996c64e4a958d8620" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46403 } }
I20260620 03:10:55.029368 21738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:55.029475 21738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a89ef086b54fe996c64e4a958d8620, State: Initialized, Role: FOLLOWER
I20260620 03:10:55.030078 21738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a89ef086b54fe996c64e4a958d8620" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46403 } }
I20260620 03:10:55.030319 21738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:10:55.030472 21738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:10:55.030676 21738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:55.032919 21738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a89ef086b54fe996c64e4a958d8620" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46403 } }
I20260620 03:10:55.033399 21738 leader_election.cc:302] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47a89ef086b54fe996c64e4a958d8620; no voters:
I20260620 03:10:55.033969 21738 leader_election.cc:288] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:10:55.034096 21748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:55.035465 21748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620 [term 1 LEADER]: Becoming Leader. State: Replica: 47a89ef086b54fe996c64e4a958d8620, State: Running, Role: LEADER
I20260620 03:10:55.035935 21748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a89ef086b54fe996c64e4a958d8620" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46403 } }
I20260620 03:10:55.036235 21738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:10:55.038494 21750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47a89ef086b54fe996c64e4a958d8620. Latest consensus state: current_term: 1 leader_uuid: "47a89ef086b54fe996c64e4a958d8620" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a89ef086b54fe996c64e4a958d8620" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46403 } } }
I20260620 03:10:55.039204 21750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:55.039414 21749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47a89ef086b54fe996c64e4a958d8620" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a89ef086b54fe996c64e4a958d8620" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46403 } } }
I20260620 03:10:55.039711 21749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620 [sys.catalog]: This master's current role is: LEADER
W20260620 03:10:55.050305 21764 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 03:10:55.050434 21764 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 03:10:55.050562 21758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:10:55.052300 21758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:10:55.060304 21758 catalog_manager.cc:1347] Generated new cluster ID: dec7bb63943f4c1d8d80b766c0776b54
I20260620 03:10:55.060418 21758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:10:55.084638 21758 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:10:55.085040 21758 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:10:55.096840 21758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47a89ef086b54fe996c64e4a958d8620: Generated new TSK 0
I20260620 03:10:55.097816 21758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:10:55.112444 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 21671
W20260620 03:10:55.113464 21733 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.102:59767
W20260620 03:10:55.114138 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 21671: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:10:55.114293 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 21671
2026-06-20T03:10:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1877 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:10:55 dist-test-slave-xv68 krb5kdc[21769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:10:55 dist-test-slave-xv68 krb5kdc[21769](info): set up 2 sockets
Jun 20 03:10:55 dist-test-slave-xv68 krb5kdc[21769](info): commencing operation
krb5kdc: starting...
W20260620 03:10:56.227694 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:10:56 dist-test-slave-xv68 krb5kdc[21769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:10:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:10:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:10:56.467706 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32949
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:56.802222 21782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:56.802531 21782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:56.802699 21782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:56.809832 21782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:10:56.809976 21782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:56.810078 21782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:56.810195 21782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:10:56.810283 21782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:10:56.820039 21782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32949
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:56.821288 21782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:56.822512 21782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:56.834160 21788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:56.834205 21787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:56.834362 21790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:56.835054 21782 server_base.cc:1034] running on GCE node
I20260620 03:10:56.835879 21782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:56.837222 21782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:56.838434 21782 hybrid_clock.cc:648] HybridClock initialized: now 1781925056838363 us; error 47 us; skew 500 ppm
I20260620 03:10:56.838769 21782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:56 dist-test-slave-xv68 krb5kdc[21769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:56.841764 21782 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:10:56.843128 21782 webserver.cc:460] Webserver started at http://127.0.0.1:43579/ using document root <none> and password file <none>
I20260620 03:10:56.843794 21782 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:56.843892 21782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:56.844129 21782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:56.846850 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "cbdaa4965f0941a78f51327ebf031a53"
format_stamp: "Formatted at 2026-06-20 03:10:56 on dist-test-slave-xv68"
server_key: "29828c1226e2cbae56fe5271df7daecc"
server_key_iv: "4f35cef34e13c01f6d12ff6de8b59c5b"
server_key_version: "encryptionkey@0"
I20260620 03:10:56.847585 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "cbdaa4965f0941a78f51327ebf031a53"
format_stamp: "Formatted at 2026-06-20 03:10:56 on dist-test-slave-xv68"
server_key: "29828c1226e2cbae56fe5271df7daecc"
server_key_iv: "4f35cef34e13c01f6d12ff6de8b59c5b"
server_key_version: "encryptionkey@0"
I20260620 03:10:56.852499 21782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 03:10:56.855715 21797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:56.856925 21782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 03:10:56.857054 21782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "cbdaa4965f0941a78f51327ebf031a53"
format_stamp: "Formatted at 2026-06-20 03:10:56 on dist-test-slave-xv68"
server_key: "29828c1226e2cbae56fe5271df7daecc"
server_key_iv: "4f35cef34e13c01f6d12ff6de8b59c5b"
server_key_version: "encryptionkey@0"
I20260620 03:10:56.857208 21782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:56.884752 21782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:56.888571 21782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:56.888864 21782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:56.907151 21782 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32949
I20260620 03:10:56.907197 21848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32949 every 8 connection(s)
I20260620 03:10:56.908490 21782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:10:56.909438 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 21782
I20260620 03:10:56.909624 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:10:56.909992 388 external_mini_cluster.cc:1428] Setting key 03a8a6380cc8e1847cd4785bf55784e6
I20260620 03:10:56.913381 21849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 03:10:56 dist-test-slave-xv68 krb5kdc[21769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 03:10:56.926514 21849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53: Bootstrap starting.
I20260620 03:10:56.930672 21849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:56.931893 21849 log.cc:824] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:56.934957 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:56.911831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:58291 (local address 127.0.0.1:32949)
0620 03:10:56.912624 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:56.912651 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:56.912705 (+ 54us) server_negotiation.cc:407] Connection header received
0620 03:10:56.915193 (+ 2488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:56.915220 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:56.915763 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:56.916200 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:56.917516 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:56.919821 (+ 2305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:56.921060 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:56.922266 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:56.922433 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:56.926387 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:56.926398 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:56.926416 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:56.926455 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:56.930376 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:56.931193 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:56.931200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:56.931207 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:56.931427 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:56.931837 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:56.931845 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:56.931848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:56.932533 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:56.932863 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:56.933390 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 03:10:56.933712 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":311,"threads_started":1}
I20260620 03:10:56.935338 21849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53: No bootstrap required, opened a new log
I20260620 03:10:56.938761 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:10:56.945318 21849 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbdaa4965f0941a78f51327ebf031a53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32949 } }
I20260620 03:10:56.945658 21849 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:56.945760 21849 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbdaa4965f0941a78f51327ebf031a53, State: Initialized, Role: FOLLOWER
I20260620 03:10:56.946389 21849 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbdaa4965f0941a78f51327ebf031a53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32949 } }
I20260620 03:10:56.946637 21849 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:10:56.946784 21849 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:10:56.946972 21849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:56.949986 21849 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbdaa4965f0941a78f51327ebf031a53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32949 } }
I20260620 03:10:56.950587 21849 leader_election.cc:302] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbdaa4965f0941a78f51327ebf031a53; no voters:
I20260620 03:10:56.951232 21849 leader_election.cc:288] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:10:56.951361 21859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:56.953183 21849 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:10:56.954473 21859 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53 [term 1 LEADER]: Becoming Leader. State: Replica: cbdaa4965f0941a78f51327ebf031a53, State: Running, Role: LEADER
I20260620 03:10:56.955179 21859 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbdaa4965f0941a78f51327ebf031a53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32949 } }
I20260620 03:10:56.959070 21861 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbdaa4965f0941a78f51327ebf031a53. Latest consensus state: current_term: 1 leader_uuid: "cbdaa4965f0941a78f51327ebf031a53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbdaa4965f0941a78f51327ebf031a53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32949 } } }
I20260620 03:10:56.959261 21860 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbdaa4965f0941a78f51327ebf031a53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbdaa4965f0941a78f51327ebf031a53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32949 } } }
I20260620 03:10:56.959626 21861 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:56.959625 21860 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53 [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:56.960914 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:56.943873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:59707 (local address 127.0.0.1:32949)
0620 03:10:56.944047 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:56.944052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:56.944396 (+ 344us) server_negotiation.cc:407] Connection header received
0620 03:10:56.944519 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:56.944524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:56.944594 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:56.944704 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:56.946111 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:56.947035 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:56.948814 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:56.949247 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:56.949338 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:56.952379 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:56.952401 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:56.952409 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:10:56.952460 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:56.956649 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:56.957772 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:56.957778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:56.957782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:56.957989 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:10:56.958660 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:10:56.958667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:10:56.958672 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:10:56.959229 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:56.959652 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:56.960360 (+ 708us) server_negotiation.cc:299] Negotiation successful
0620 03:10:56.960555 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 03:10:56.968317 21868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:10:56.972662 21868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:10:56.981184 21868 catalog_manager.cc:1347] Generated new cluster ID: 810574c4632e4ad9b4fb196c3282ae56
I20260620 03:10:56.981298 21868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:10:56.993345 21868 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:10:56.994539 21868 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:10:57.003788 21868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbdaa4965f0941a78f51327ebf031a53: Generated new TSK 0
I20260620 03:10:57.004773 21868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:10:57.012498 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 21782
I20260620 03:10:57.014014 21844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:10:57.104287 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 21782
W20260620 03:10:57.121930 21767 connection.cc:535] client connection to 127.0.0.1:32949 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32949): Connection reset by peer (error 104)
2026-06-20T03:10:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1990 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:10:57 dist-test-slave-xv68 krb5kdc[21881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:10:57 dist-test-slave-xv68 krb5kdc[21881](info): set up 2 sockets
Jun 20 03:10:57 dist-test-slave-xv68 krb5kdc[21881](info): commencing operation
krb5kdc: starting...
W20260620 03:10:58.195715 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:10:58 dist-test-slave-xv68 krb5kdc[21881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:10:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:10:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:10:58.446273 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33117
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:10:58.765964 21894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:10:58.766229 21894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:10:58.766394 21894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:10:58.773492 21894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:10:58.773597 21894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:10:58.773710 21894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:10:58.773810 21894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:10:58.773890 21894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:10:58.783514 21894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33117
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:10:58.784765 21894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:10:58.786062 21894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:10:58.797888 21899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:58.797845 21902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:10:58.797925 21900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:10:58.798745 21894 server_base.cc:1034] running on GCE node
I20260620 03:10:58.799573 21894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:10:58.800904 21894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:10:58.802155 21894 hybrid_clock.cc:648] HybridClock initialized: now 1781925058802130 us; error 66 us; skew 500 ppm
I20260620 03:10:58.802498 21894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:10:58 dist-test-slave-xv68 krb5kdc[21881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:10:58.805495 21894 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:10:58.806990 21894 webserver.cc:460] Webserver started at http://127.0.0.1:43197/ using document root <none> and password file <none>
I20260620 03:10:58.807749 21894 fs_manager.cc:359] Metadata directory not provided
I20260620 03:10:58.807898 21894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:10:58.808207 21894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:10:58.811089 21894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "ce8d71b446684933b8feb4fb330eb98d"
format_stamp: "Formatted at 2026-06-20 03:10:58 on dist-test-slave-xv68"
server_key: "f87409ff0e014ef7c4131470b45b3f37"
server_key_iv: "706f2ea1643ef77a8c7cf0576fe8c09d"
server_key_version: "encryptionkey@0"
I20260620 03:10:58.811928 21894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "ce8d71b446684933b8feb4fb330eb98d"
format_stamp: "Formatted at 2026-06-20 03:10:58 on dist-test-slave-xv68"
server_key: "f87409ff0e014ef7c4131470b45b3f37"
server_key_iv: "706f2ea1643ef77a8c7cf0576fe8c09d"
server_key_version: "encryptionkey@0"
I20260620 03:10:58.816869 21894 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:10:58.820191 21909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:10:58.821453 21894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:10:58.821641 21894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "ce8d71b446684933b8feb4fb330eb98d"
format_stamp: "Formatted at 2026-06-20 03:10:58 on dist-test-slave-xv68"
server_key: "f87409ff0e014ef7c4131470b45b3f37"
server_key_iv: "706f2ea1643ef77a8c7cf0576fe8c09d"
server_key_version: "encryptionkey@0"
I20260620 03:10:58.821846 21894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:10:58.838413 21894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:10:58.842437 21894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:10:58.842718 21894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:10:58.861440 21894 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33117
I20260620 03:10:58.861475 21960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33117 every 8 connection(s)
I20260620 03:10:58.862739 21894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:10:58.866374 21961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:10:58.869900 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 21894
I20260620 03:10:58.870077 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:10:58.870626 388 external_mini_cluster.cc:1428] Setting key d25e23d5242b64ddee393e5a9e71151d
I20260620 03:10:58.878464 21961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d: Bootstrap starting.
I20260620 03:10:58.882019 21961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d: Neither blocks nor log segments found. Creating new log.
I20260620 03:10:58.883092 21961 log.cc:824] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d: Log is configured to *not* fsync() on all Append() calls
I20260620 03:10:58.883617 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:58.872346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:52135 (local address 127.0.0.1:33117)
0620 03:10:58.873061 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:58.873071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:58.873107 (+ 36us) server_negotiation.cc:407] Connection header received
0620 03:10:58.874677 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:58.874696 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:58.875097 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:58.875506 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:58.876005 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:58.877335 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:58.879175 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:58.879877 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:58.880012 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:58.881257 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:58.881264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:58.881276 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:58.881773 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:58.881842 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:58.881865 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:58.881977 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:58.882403 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 03:10:58.882571 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":336,"threads_started":1}
I20260620 03:10:58.886286 21961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d: No bootstrap required, opened a new log
I20260620 03:10:58.887636 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:10:58.895093 21961 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce8d71b446684933b8feb4fb330eb98d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } }
I20260620 03:10:58.895519 21961 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:10:58.895632 21961 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce8d71b446684933b8feb4fb330eb98d, State: Initialized, Role: FOLLOWER
I20260620 03:10:58.896260 21961 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce8d71b446684933b8feb4fb330eb98d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } }
I20260620 03:10:58.896548 21961 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:10:58.896685 21961 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:10:58.896879 21961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:10:58.898753 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:10:58.891840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:56727 (local address 127.0.0.1:33117)
0620 03:10:58.892017 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 03:10:58.892023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:10:58.892364 (+ 341us) server_negotiation.cc:407] Connection header received
0620 03:10:58.892499 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:10:58.892504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:10:58.892575 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:10:58.892683 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:10:58.893220 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:58.894381 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:58.896019 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:10:58.896702 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:10:58.896850 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:10:58.897973 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:10:58.897983 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:10:58.897988 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:10:58.898008 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:10:58.898072 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:10:58.898092 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:10:58.898223 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 03:10:58.898380 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 03:10:58.898428 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:10:58.900115 21961 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce8d71b446684933b8feb4fb330eb98d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } }
I20260620 03:10:58.900627 21961 leader_election.cc:302] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce8d71b446684933b8feb4fb330eb98d; no voters:
I20260620 03:10:58.901161 21961 leader_election.cc:288] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:10:58.901275 21971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:10:58.902491 21971 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d [term 1 LEADER]: Becoming Leader. State: Replica: ce8d71b446684933b8feb4fb330eb98d, State: Running, Role: LEADER
I20260620 03:10:58.903086 21971 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce8d71b446684933b8feb4fb330eb98d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } }
I20260620 03:10:58.903235 21961 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:10:58.905850 21972 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce8d71b446684933b8feb4fb330eb98d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce8d71b446684933b8feb4fb330eb98d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } } }
I20260620 03:10:58.906275 21972 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:58.906821 21973 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce8d71b446684933b8feb4fb330eb98d. Latest consensus state: current_term: 1 leader_uuid: "ce8d71b446684933b8feb4fb330eb98d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce8d71b446684933b8feb4fb330eb98d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } } }
I20260620 03:10:58.907109 21973 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d [sys.catalog]: This master's current role is: LEADER
I20260620 03:10:58.914161 21980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:10:58.919128 21980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:10:58.928011 21980 catalog_manager.cc:1347] Generated new cluster ID: 18430f7be7ac4d628acdb925b31455b7
I20260620 03:10:58.928112 21980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:10:58.943194 21980 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:10:58.944515 21980 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:10:58.955679 21980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce8d71b446684933b8feb4fb330eb98d: Generated new TSK 0
I20260620 03:10:58.956547 21980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:10:58.961681 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 21894
W20260620 03:10:58.962838 21956 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.102:52135
W20260620 03:10:58.963584 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 21894: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:10:58.963728 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 21894
W20260620 03:10:58.977511 21879 connection.cc:535] client connection to 127.0.0.1:33117 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33117): Connection reset by peer (error 104)
2026-06-20T03:10:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1855 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:10:59 dist-test-slave-xv68 krb5kdc[21992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:10:59 dist-test-slave-xv68 krb5kdc[21992](info): set up 2 sockets
Jun 20 03:10:59 dist-test-slave-xv68 krb5kdc[21992](info): commencing operation
krb5kdc: starting...
W20260620 03:11:00.067739 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:11:00 dist-test-slave-xv68 krb5kdc[21992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:11:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:11:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:11:00.304177 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:11:00.624177 22005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:11:00.624445 22005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:11:00.624655 22005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:11:00.631623 22005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:11:00.631749 22005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:11:00.631837 22005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:11:00.631949 22005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:11:00.632032 22005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:11:00.641779 22005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:11:00.643021 22005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:11:00.644268 22005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:11:00.655970 22013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:00.656116 22011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:00.656091 22010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:11:00.656884 22005 server_base.cc:1034] running on GCE node
I20260620 03:11:00.657599 22005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:11:00.658977 22005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:11:00.660199 22005 hybrid_clock.cc:648] HybridClock initialized: now 1781925060660156 us; error 42 us; skew 500 ppm
I20260620 03:11:00.660540 22005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:11:00 dist-test-slave-xv68 krb5kdc[21992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:11:00.667095 22005 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:11:00.668646 22005 webserver.cc:460] Webserver started at http://127.0.0.1:44317/ using document root <none> and password file <none>
I20260620 03:11:00.669317 22005 fs_manager.cc:359] Metadata directory not provided
I20260620 03:11:00.669466 22005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:11:00.669809 22005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:11:00.672767 22005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "924dbcc6f41c4849955de4ed4c3cda68"
format_stamp: "Formatted at 2026-06-20 03:11:00 on dist-test-slave-xv68"
server_key: "ed18c815885e308cc4d42e6ff1203262"
server_key_iv: "bba244d4cdd6d6d394667ac10b5912c8"
server_key_version: "encryptionkey@0"
I20260620 03:11:00.673517 22005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "924dbcc6f41c4849955de4ed4c3cda68"
format_stamp: "Formatted at 2026-06-20 03:11:00 on dist-test-slave-xv68"
server_key: "ed18c815885e308cc4d42e6ff1203262"
server_key_iv: "bba244d4cdd6d6d394667ac10b5912c8"
server_key_version: "encryptionkey@0"
I20260620 03:11:00.678403 22005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 03:11:00.681636 22020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:11:00.682811 22005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 03:11:00.682993 22005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "924dbcc6f41c4849955de4ed4c3cda68"
format_stamp: "Formatted at 2026-06-20 03:11:00 on dist-test-slave-xv68"
server_key: "ed18c815885e308cc4d42e6ff1203262"
server_key_iv: "bba244d4cdd6d6d394667ac10b5912c8"
server_key_version: "encryptionkey@0"
I20260620 03:11:00.683193 22005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:11:00.688684 22005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:11:00.692488 22005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:11:00.692763 22005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:11:00.710729 22005 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36825
I20260620 03:11:00.710810 22071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36825 every 8 connection(s)
I20260620 03:11:00.712152 22005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:11:00.715750 22072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:11:00.719919 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 22005
I20260620 03:11:00.720101 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:11:00.720624 388 external_mini_cluster.cc:1428] Setting key c732e23fa2741aa6eefe0445db0a1848
I20260620 03:11:00.727878 22075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:00.722275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:34077 (local address 127.0.0.1:36825)
0620 03:11:00.723002 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:00.723013 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:00.723050 (+ 37us) server_negotiation.cc:407] Connection header received
0620 03:11:00.724639 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:00.724659 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:00.724975 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:00.725278 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:00.726356 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:00.726372 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:00.726383 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:11:00.726400 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:00.726458 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:11:00.726475 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:00.726636 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:00.726751 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 03:11:00.726939 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":333,"threads_started":1}
I20260620 03:11:00.728653 22072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68: Bootstrap starting.
I20260620 03:11:00.731634 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:11:00.732373 22072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68: Neither blocks nor log segments found. Creating new log.
I20260620 03:11:00.733564 22072 log.cc:824] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68: Log is configured to *not* fsync() on all Append() calls
I20260620 03:11:00.736150 22072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68: No bootstrap required, opened a new log
I20260620 03:11:00.739225 22075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:00.736722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:43553 (local address 127.0.0.1:36825)
0620 03:11:00.736934 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:00.736943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:00.737387 (+ 444us) server_negotiation.cc:407] Connection header received
0620 03:11:00.737572 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:00.737576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:00.737620 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:00.737704 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:00.738586 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:00.738590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:00.738593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:11:00.738605 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:00.738634 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:11:00.738647 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:00.738738 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:00.738864 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 03:11:00.738914 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 03:11:00.744603 22072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924dbcc6f41c4849955de4ed4c3cda68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } }
I20260620 03:11:00.744954 22072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:11:00.745065 22072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924dbcc6f41c4849955de4ed4c3cda68, State: Initialized, Role: FOLLOWER
I20260620 03:11:00.745676 22072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924dbcc6f41c4849955de4ed4c3cda68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } }
I20260620 03:11:00.745913 22072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:11:00.746054 22072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:11:00.746227 22072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:11:00.748497 22072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924dbcc6f41c4849955de4ed4c3cda68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } }
I20260620 03:11:00.748958 22072 leader_election.cc:302] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 924dbcc6f41c4849955de4ed4c3cda68; no voters:
I20260620 03:11:00.749533 22072 leader_election.cc:288] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:11:00.749661 22082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:11:00.750329 22082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68 [term 1 LEADER]: Becoming Leader. State: Replica: 924dbcc6f41c4849955de4ed4c3cda68, State: Running, Role: LEADER
I20260620 03:11:00.750861 22082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924dbcc6f41c4849955de4ed4c3cda68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } }
I20260620 03:11:00.751645 22072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:11:00.753646 22083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "924dbcc6f41c4849955de4ed4c3cda68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924dbcc6f41c4849955de4ed4c3cda68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } } }
I20260620 03:11:00.754019 22083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68 [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:00.754472 22084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 924dbcc6f41c4849955de4ed4c3cda68. Latest consensus state: current_term: 1 leader_uuid: "924dbcc6f41c4849955de4ed4c3cda68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924dbcc6f41c4849955de4ed4c3cda68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } } }
I20260620 03:11:00.754737 22084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68 [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:00.754978 22089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:11:00.760659 22089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:11:00.768951 22089 catalog_manager.cc:1347] Generated new cluster ID: 0fdfcbfebe594f1b86ee87cc32f2f1f1
I20260620 03:11:00.769068 22089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:11:00.786887 22089 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:11:00.787356 22089 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:11:00.799263 22089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 924dbcc6f41c4849955de4ed4c3cda68: Generated new TSK 0
I20260620 03:11:00.800331 22089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:11:00.812069 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 22005
W20260620 03:11:00.813078 22067 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.102:34077
W20260620 03:11:00.813735 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 22005: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:11:00.813895 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 22005
2026-06-20T03:11:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1853 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:11:00 dist-test-slave-xv68 krb5kdc[22103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:11:00 dist-test-slave-xv68 krb5kdc[22103](info): set up 2 sockets
Jun 20 03:11:00 dist-test-slave-xv68 krb5kdc[22103](info): commencing operation
krb5kdc: starting...
W20260620 03:11:01.923707 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:11:02 dist-test-slave-xv68 krb5kdc[22103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:11:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:11:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:11:02.164641 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44079
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:11:02.485029 22116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:11:02.485309 22116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:11:02.485498 22116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:11:02.492662 22116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:11:02.492775 22116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:11:02.492888 22116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:11:02.492970 22116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:11:02.493049 22116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:11:02.502766 22116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44079
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:11:02.504112 22116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:11:02.505421 22116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:11:02.516978 22121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:02.516937 22122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:02.517042 22124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:11:02.517761 22116 server_base.cc:1034] running on GCE node
I20260620 03:11:02.518515 22116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:11:02.519958 22116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:11:02.521155 22116 hybrid_clock.cc:648] HybridClock initialized: now 1781925062521124 us; error 29 us; skew 500 ppm
I20260620 03:11:02.521517 22116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:11:02 dist-test-slave-xv68 krb5kdc[22103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:11:02.524590 22116 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:11:02.525965 22116 webserver.cc:460] Webserver started at http://127.0.97.62:38117/ using document root <none> and password file <none>
I20260620 03:11:02.526611 22116 fs_manager.cc:359] Metadata directory not provided
I20260620 03:11:02.526741 22116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:11:02.527046 22116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:11:02.529978 22116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "e558a6213bee48bf8caf52c5acc2bcba"
format_stamp: "Formatted at 2026-06-20 03:11:02 on dist-test-slave-xv68"
server_key: "86a201c569fe52f75029b8c13a59b81b"
server_key_iv: "f2378bd0ca08c35911ef938d14748dbf"
server_key_version: "encryptionkey@0"
I20260620 03:11:02.530709 22116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "e558a6213bee48bf8caf52c5acc2bcba"
format_stamp: "Formatted at 2026-06-20 03:11:02 on dist-test-slave-xv68"
server_key: "86a201c569fe52f75029b8c13a59b81b"
server_key_iv: "f2378bd0ca08c35911ef938d14748dbf"
server_key_version: "encryptionkey@0"
I20260620 03:11:02.535730 22116 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260620 03:11:02.539162 22131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:11:02.540596 22116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:11:02.540771 22116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "e558a6213bee48bf8caf52c5acc2bcba"
format_stamp: "Formatted at 2026-06-20 03:11:02 on dist-test-slave-xv68"
server_key: "86a201c569fe52f75029b8c13a59b81b"
server_key_iv: "f2378bd0ca08c35911ef938d14748dbf"
server_key_version: "encryptionkey@0"
I20260620 03:11:02.540990 22116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:11:02.567071 22116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:11:02.571408 22116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:11:02.571705 22116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:11:02.590794 22116 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44079
I20260620 03:11:02.590837 22182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44079 every 8 connection(s)
I20260620 03:11:02.592224 22116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:11:02.595947 22183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:11:02.597463 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 22116
I20260620 03:11:02.597647 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:11:02.598179 388 external_mini_cluster.cc:1428] Setting key ac882bef43d478dd7a0392eb10739231
I20260620 03:11:02.610623 22183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba: Bootstrap starting.
Jun 20 03:11:02 dist-test-slave-xv68 krb5kdc[22103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:11:02.614162 22183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba: Neither blocks nor log segments found. Creating new log.
I20260620 03:11:02.615274 22183 log.cc:824] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba: Log is configured to *not* fsync() on all Append() calls
I20260620 03:11:02.618175 22183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba: No bootstrap required, opened a new log
I20260620 03:11:02.620976 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:02.600020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42579 (local address 127.0.97.62:44079)
0620 03:11:02.600809 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:02.600821 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:02.600862 (+ 41us) server_negotiation.cc:407] Connection header received
0620 03:11:02.602496 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:02.602520 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:02.602918 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:02.603245 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:02.604920 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:02.606200 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:02.608119 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:02.608776 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:02.608918 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:02.613685 (+ 4767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:02.613693 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:02.613707 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:02.613743 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:02.616902 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:02.617679 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:02.617685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:02.617691 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:02.617850 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:02.618334 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:02.618337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:02.618339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:02.618826 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:02.619183 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:02.619676 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 03:11:02.620006 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":378,"threads_started":1}
I20260620 03:11:02.625007 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:11:02.626852 22183 raft_consensus.cc:348] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e558a6213bee48bf8caf52c5acc2bcba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44079 } }
I20260620 03:11:02.627234 22183 raft_consensus.cc:374] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:11:02.627383 22183 raft_consensus.cc:729] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e558a6213bee48bf8caf52c5acc2bcba, State: Initialized, Role: FOLLOWER
I20260620 03:11:02.628014 22183 consensus_queue.cc:260] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e558a6213bee48bf8caf52c5acc2bcba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44079 } }
I20260620 03:11:02.628259 22183 raft_consensus.cc:388] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:11:02.628412 22183 raft_consensus.cc:482] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:11:02.628590 22183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:11:02.631548 22183 raft_consensus.cc:504] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e558a6213bee48bf8caf52c5acc2bcba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44079 } }
I20260620 03:11:02.632170 22183 leader_election.cc:302] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e558a6213bee48bf8caf52c5acc2bcba; no voters:
I20260620 03:11:02.632818 22183 leader_election.cc:288] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:11:02.632920 22193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:11:02.634176 22193 raft_consensus.cc:686] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba [term 1 LEADER]: Becoming Leader. State: Replica: e558a6213bee48bf8caf52c5acc2bcba, State: Running, Role: LEADER
I20260620 03:11:02.634925 22183 sys_catalog.cc:564] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:11:02.634934 22193 consensus_queue.cc:237] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e558a6213bee48bf8caf52c5acc2bcba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44079 } }
I20260620 03:11:02.641222 22194 sys_catalog.cc:455] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e558a6213bee48bf8caf52c5acc2bcba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e558a6213bee48bf8caf52c5acc2bcba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44079 } } }
I20260620 03:11:02.641621 22194 sys_catalog.cc:458] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:02.642190 22195 sys_catalog.cc:455] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba [sys.catalog]: SysCatalogTable state changed. Reason: New leader e558a6213bee48bf8caf52c5acc2bcba. Latest consensus state: current_term: 1 leader_uuid: "e558a6213bee48bf8caf52c5acc2bcba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e558a6213bee48bf8caf52c5acc2bcba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44079 } } }
I20260620 03:11:02.642541 22195 sys_catalog.cc:458] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:02.650280 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:02.629408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39561 (local address 127.0.97.62:44079)
0620 03:11:02.629618 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:02.629625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:02.630023 (+ 398us) server_negotiation.cc:407] Connection header received
0620 03:11:02.630159 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:02.630164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:02.630238 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:02.630350 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:02.631709 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:02.632903 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:02.634936 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:02.635642 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:02.635744 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:02.639810 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:02.639824 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:02.639832 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:02.639896 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:02.645226 (+ 5330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:02.646356 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:02.646362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:02.646365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:02.646502 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:02.647154 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:02.647162 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:02.647168 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:02.647727 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:02.648095 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:02.649516 (+ 1421us) server_negotiation.cc:299] Negotiation successful
0620 03:11:02.649753 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 03:11:02.661114 22201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:11:02.665663 22201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:11:02.673719 22201 catalog_manager.cc:1347] Generated new cluster ID: 23deb76cd2d644c89f6e57e59f7ebdc8
I20260620 03:11:02.673856 22201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:11:02.698325 22201 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:11:02.699721 22201 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:11:02.707857 22201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e558a6213bee48bf8caf52c5acc2bcba: Generated new TSK 0
I20260620 03:11:02.708688 22201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:11:02.721096 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 22116
I20260620 03:11:02.722226 22178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:11:02.815359 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 22116
W20260620 03:11:02.832914 22101 connection.cc:535] client connection to 127.0.97.62:44079 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:44079): Connection reset by peer (error 104)
2026-06-20T03:11:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2002 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:11:02 dist-test-slave-xv68 krb5kdc[22216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:11:02 dist-test-slave-xv68 krb5kdc[22216](info): set up 2 sockets
Jun 20 03:11:02 dist-test-slave-xv68 krb5kdc[22216](info): commencing operation
krb5kdc: starting...
W20260620 03:11:03.923651 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:11:04 dist-test-slave-xv68 krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:11:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:11:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:11:04.164499 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45591
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:11:04.489486 22229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:11:04.489748 22229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:11:04.489837 22229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:11:04.496901 22229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:11:04.496995 22229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:11:04.497056 22229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:11:04.497108 22229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:11:04.497150 22229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:11:04.506789 22229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45591
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:11:04.508028 22229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:11:04.509217 22229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:11:04.520742 22235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:04.520784 22234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:04.520695 22237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:11:04.521524 22229 server_base.cc:1034] running on GCE node
I20260620 03:11:04.522221 22229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:11:04.523517 22229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:11:04.524731 22229 hybrid_clock.cc:648] HybridClock initialized: now 1781925064524692 us; error 54 us; skew 500 ppm
I20260620 03:11:04.525061 22229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:11:04 dist-test-slave-xv68 krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:11:04.528049 22229 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:11:04.529372 22229 webserver.cc:460] Webserver started at http://127.0.97.62:32839/ using document root <none> and password file <none>
I20260620 03:11:04.529994 22229 fs_manager.cc:359] Metadata directory not provided
I20260620 03:11:04.530130 22229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:11:04.530409 22229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:11:04.533231 22229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "7ed5afe79d7542c7b9f84eb1a9fdfd83"
format_stamp: "Formatted at 2026-06-20 03:11:04 on dist-test-slave-xv68"
server_key: "7baf17dd1037e969e971da5c0a2ac157"
server_key_iv: "3b247441bffd9ba4f782475387d12b05"
server_key_version: "encryptionkey@0"
I20260620 03:11:04.533942 22229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "7ed5afe79d7542c7b9f84eb1a9fdfd83"
format_stamp: "Formatted at 2026-06-20 03:11:04 on dist-test-slave-xv68"
server_key: "7baf17dd1037e969e971da5c0a2ac157"
server_key_iv: "3b247441bffd9ba4f782475387d12b05"
server_key_version: "encryptionkey@0"
I20260620 03:11:04.538787 22229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260620 03:11:04.541965 22244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:11:04.543130 22229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 03:11:04.543326 22229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "7ed5afe79d7542c7b9f84eb1a9fdfd83"
format_stamp: "Formatted at 2026-06-20 03:11:04 on dist-test-slave-xv68"
server_key: "7baf17dd1037e969e971da5c0a2ac157"
server_key_iv: "3b247441bffd9ba4f782475387d12b05"
server_key_version: "encryptionkey@0"
I20260620 03:11:04.543536 22229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:11:04.565878 22229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:11:04.569818 22229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:11:04.570091 22229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:11:04.588438 22229 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45591
I20260620 03:11:04.588490 22295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45591 every 8 connection(s)
I20260620 03:11:04.589730 22229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:11:04.593158 22296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:11:04.598049 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 22229
I20260620 03:11:04.598223 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:11:04.598670 388 external_mini_cluster.cc:1428] Setting key 51853df73a1dc343c35bf0762000eb7d
I20260620 03:11:04.603474 22296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83: Bootstrap starting.
I20260620 03:11:04.606984 22296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83: Neither blocks nor log segments found. Creating new log.
I20260620 03:11:04.608139 22296 log.cc:824] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83: Log is configured to *not* fsync() on all Append() calls
I20260620 03:11:04.611022 22296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83: No bootstrap required, opened a new log
I20260620 03:11:04.611868 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:04.600479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55105 (local address 127.0.97.62:45591)
0620 03:11:04.601320 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:04.601331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:04.601369 (+ 38us) server_negotiation.cc:407] Connection header received
0620 03:11:04.602959 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:04.602982 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:04.603407 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:04.603774 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:04.604347 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:04.605648 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:04.607517 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:04.608148 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:04.608309 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:04.609643 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:04.609651 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:04.609664 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:11:04.610160 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:04.610228 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:11:04.610251 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:04.610368 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:04.610654 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 03:11:04.610847 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":408,"threads_started":1}
I20260620 03:11:04.615696 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:11:04.619441 22296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed5afe79d7542c7b9f84eb1a9fdfd83" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45591 } }
I20260620 03:11:04.619863 22296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:11:04.619987 22296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed5afe79d7542c7b9f84eb1a9fdfd83, State: Initialized, Role: FOLLOWER
I20260620 03:11:04.620672 22296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed5afe79d7542c7b9f84eb1a9fdfd83" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45591 } }
I20260620 03:11:04.620956 22296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:11:04.621114 22296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:11:04.621273 22296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:11:04.624015 22296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed5afe79d7542c7b9f84eb1a9fdfd83" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45591 } }
I20260620 03:11:04.624562 22296 leader_election.cc:302] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ed5afe79d7542c7b9f84eb1a9fdfd83; no voters:
I20260620 03:11:04.625203 22296 leader_election.cc:288] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:11:04.625385 22306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:11:04.626060 22306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83 [term 1 LEADER]: Becoming Leader. State: Replica: 7ed5afe79d7542c7b9f84eb1a9fdfd83, State: Running, Role: LEADER
I20260620 03:11:04.626873 22306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed5afe79d7542c7b9f84eb1a9fdfd83" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45591 } }
I20260620 03:11:04.627629 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:04.619763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46367 (local address 127.0.97.62:45591)
0620 03:11:04.619957 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:04.619963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:04.620471 (+ 508us) server_negotiation.cc:407] Connection header received
0620 03:11:04.620597 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:04.620602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:04.620685 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:04.620792 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:04.621353 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:04.622530 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:04.623907 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:04.624583 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:04.624722 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:04.626033 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:04.626049 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:04.626058 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:11:04.626080 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:04.626156 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:11:04.626185 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:04.626682 (+ 497us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:04.626770 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 03:11:04.626819 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 03:11:04.628628 22296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:11:04.630347 22308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ed5afe79d7542c7b9f84eb1a9fdfd83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed5afe79d7542c7b9f84eb1a9fdfd83" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45591 } } }
I20260620 03:11:04.630681 22308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83 [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:04.632287 22312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:11:04.632589 22307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ed5afe79d7542c7b9f84eb1a9fdfd83. Latest consensus state: current_term: 1 leader_uuid: "7ed5afe79d7542c7b9f84eb1a9fdfd83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed5afe79d7542c7b9f84eb1a9fdfd83" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45591 } } }
I20260620 03:11:04.632881 22307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83 [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:04.637275 22312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:11:04.645833 22312 catalog_manager.cc:1347] Generated new cluster ID: c2f568f6a7eb462eb6a098f4c33777d5
I20260620 03:11:04.645978 22312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:11:04.666769 22312 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:11:04.668597 22312 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:11:04.681555 22312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ed5afe79d7542c7b9f84eb1a9fdfd83: Generated new TSK 0
I20260620 03:11:04.682485 22312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:11:04.692826 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 22229
W20260620 03:11:04.694465 22291 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55105
W20260620 03:11:04.695216 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 22229: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:11:04.695385 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 22229
W20260620 03:11:04.712131 22214 connection.cc:535] client connection to 127.0.97.62:45591 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:45591): Connection reset by peer (error 104)
2026-06-20T03:11:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:11:04 dist-test-slave-xv68 krb5kdc[22327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:11:04 dist-test-slave-xv68 krb5kdc[22327](info): set up 2 sockets
Jun 20 03:11:04 dist-test-slave-xv68 krb5kdc[22327](info): commencing operation
krb5kdc: starting...
W20260620 03:11:05.811743 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:11:05 dist-test-slave-xv68 krb5kdc[22327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:11:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:11:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:11:06.064579 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36811
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:11:06.391682 22340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:11:06.391963 22340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:11:06.392170 22340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:11:06.400714 22340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:11:06.400825 22340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:11:06.400939 22340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:11:06.401034 22340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:11:06.401098 22340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:11:06.410869 22340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36811
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:11:06.412128 22340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:11:06.413391 22340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:11:06.424793 22345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:06.424885 22348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:06.424855 22346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:11:06.425756 22340 server_base.cc:1034] running on GCE node
I20260620 03:11:06.426520 22340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:11:06.427917 22340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:11:06.429165 22340 hybrid_clock.cc:648] HybridClock initialized: now 1781925066429091 us; error 54 us; skew 500 ppm
I20260620 03:11:06.429507 22340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:11:06 dist-test-slave-xv68 krb5kdc[22327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:11:06.435863 22340 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:11:06.437322 22340 webserver.cc:460] Webserver started at http://127.0.97.62:37429/ using document root <none> and password file <none>
I20260620 03:11:06.437945 22340 fs_manager.cc:359] Metadata directory not provided
I20260620 03:11:06.438046 22340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:11:06.438374 22340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:11:06.441262 22340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "1e4235c496894b7492e6d9d5be89b6bc"
format_stamp: "Formatted at 2026-06-20 03:11:06 on dist-test-slave-xv68"
server_key: "befef8451b587a7c5084799e8a0a2713"
server_key_iv: "ec22fd021d8f9432ff991a2ee9a98289"
server_key_version: "encryptionkey@0"
I20260620 03:11:06.441977 22340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "1e4235c496894b7492e6d9d5be89b6bc"
format_stamp: "Formatted at 2026-06-20 03:11:06 on dist-test-slave-xv68"
server_key: "befef8451b587a7c5084799e8a0a2713"
server_key_iv: "ec22fd021d8f9432ff991a2ee9a98289"
server_key_version: "encryptionkey@0"
I20260620 03:11:06.447028 22340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:11:06.451030 22355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:11:06.452409 22340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:11:06.452575 22340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "1e4235c496894b7492e6d9d5be89b6bc"
format_stamp: "Formatted at 2026-06-20 03:11:06 on dist-test-slave-xv68"
server_key: "befef8451b587a7c5084799e8a0a2713"
server_key_iv: "ec22fd021d8f9432ff991a2ee9a98289"
server_key_version: "encryptionkey@0"
I20260620 03:11:06.452744 22340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:11:06.458535 22340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:11:06.462567 22340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:11:06.462808 22340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:11:06.481317 22340 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36811
I20260620 03:11:06.481354 22406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36811 every 8 connection(s)
I20260620 03:11:06.482618 22340 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:11:06.486091 22407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:11:06.489327 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 22340
I20260620 03:11:06.489499 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:11:06.490031 388 external_mini_cluster.cc:1428] Setting key 94d4d26f317250567aae53b4a0200d39
I20260620 03:11:06.497534 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:06.491853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56129 (local address 127.0.97.62:36811)
0620 03:11:06.492588 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:06.492600 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:06.492638 (+ 38us) server_negotiation.cc:407] Connection header received
0620 03:11:06.494318 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:06.494338 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:06.494660 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:06.495068 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:06.496077 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:06.496083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:06.496096 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:11:06.496112 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:06.496169 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:11:06.496187 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:06.496351 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:06.496455 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 03:11:06.496659 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":308,"threads_started":1}
I20260620 03:11:06.498410 22407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc: Bootstrap starting.
I20260620 03:11:06.500931 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:11:06.501645 22407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc: Neither blocks nor log segments found. Creating new log.
I20260620 03:11:06.502830 22407 log.cc:824] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc: Log is configured to *not* fsync() on all Append() calls
I20260620 03:11:06.505383 22407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc: No bootstrap required, opened a new log
I20260620 03:11:06.508570 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:06.506050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56463 (local address 127.0.97.62:36811)
0620 03:11:06.506237 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:06.506243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:06.506635 (+ 392us) server_negotiation.cc:407] Connection header received
0620 03:11:06.506755 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:06.506761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:06.506821 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:06.506930 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:06.507841 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:06.507847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:06.507852 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:11:06.507870 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:06.507918 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:11:06.507937 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:06.508068 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:06.508204 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 03:11:06.508257 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 03:11:06.513777 22407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4235c496894b7492e6d9d5be89b6bc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36811 } }
I20260620 03:11:06.514278 22407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:11:06.514397 22407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4235c496894b7492e6d9d5be89b6bc, State: Initialized, Role: FOLLOWER
I20260620 03:11:06.514971 22407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4235c496894b7492e6d9d5be89b6bc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36811 } }
I20260620 03:11:06.515201 22407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:11:06.515369 22407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:11:06.515555 22407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:11:06.517628 22407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4235c496894b7492e6d9d5be89b6bc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36811 } }
I20260620 03:11:06.518093 22407 leader_election.cc:302] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e4235c496894b7492e6d9d5be89b6bc; no voters:
I20260620 03:11:06.518718 22407 leader_election.cc:288] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:11:06.518867 22417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:11:06.519593 22417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc [term 1 LEADER]: Becoming Leader. State: Replica: 1e4235c496894b7492e6d9d5be89b6bc, State: Running, Role: LEADER
I20260620 03:11:06.520167 22417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4235c496894b7492e6d9d5be89b6bc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36811 } }
I20260620 03:11:06.521013 22407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:11:06.523336 22419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e4235c496894b7492e6d9d5be89b6bc. Latest consensus state: current_term: 1 leader_uuid: "1e4235c496894b7492e6d9d5be89b6bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4235c496894b7492e6d9d5be89b6bc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36811 } } }
I20260620 03:11:06.523464 22418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e4235c496894b7492e6d9d5be89b6bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4235c496894b7492e6d9d5be89b6bc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36811 } } }
I20260620 03:11:06.523766 22419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:06.523841 22418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:06.525226 22422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:11:06.530735 22422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:11:06.538825 22422 catalog_manager.cc:1347] Generated new cluster ID: f0aa5bc57fce4796af115fad29301325
I20260620 03:11:06.538944 22422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:11:06.554853 22422 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:11:06.555359 22422 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:11:06.568650 22422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e4235c496894b7492e6d9d5be89b6bc: Generated new TSK 0
I20260620 03:11:06.569609 22422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:11:06.574427 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 22340
W20260620 03:11:06.575635 22402 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56129
W20260620 03:11:06.576308 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 22340: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:11:06.576453 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 22340
2026-06-20T03:11:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1881 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:11:06 dist-test-slave-xv68 krb5kdc[22438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:11:06 dist-test-slave-xv68 krb5kdc[22438](info): set up 2 sockets
Jun 20 03:11:06 dist-test-slave-xv68 krb5kdc[22438](info): commencing operation
krb5kdc: starting...
W20260620 03:11:07.691732 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:11:07 dist-test-slave-xv68 krb5kdc[22438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:11:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:11:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:11:07.932451 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36741
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:11:08.262758 22451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:11:08.263022 22451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:11:08.263216 22451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:11:08.270206 22451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:11:08.270304 22451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:11:08.270352 22451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:11:08.270462 22451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:11:08.270532 22451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:11:08.280371 22451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:11:08.281615 22451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:11:08.282912 22451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:11:08.295161 22456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:08.295214 22457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:08.295383 22459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:11:08.296363 22451 server_base.cc:1034] running on GCE node
I20260620 03:11:08.297061 22451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:11:08.298324 22451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:11:08.299568 22451 hybrid_clock.cc:648] HybridClock initialized: now 1781925068299522 us; error 29 us; skew 500 ppm
I20260620 03:11:08.299873 22451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:11:08 dist-test-slave-xv68 krb5kdc[22438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:11:08.302948 22451 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:11:08.304384 22451 webserver.cc:460] Webserver started at http://127.0.97.62:43141/ using document root <none> and password file <none>
I20260620 03:11:08.305008 22451 fs_manager.cc:359] Metadata directory not provided
I20260620 03:11:08.305110 22451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:11:08.305431 22451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:11:08.308352 22451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "7ce4594f2523402497677d37b5f8eca9"
format_stamp: "Formatted at 2026-06-20 03:11:08 on dist-test-slave-xv68"
server_key: "718b1e21dda634fc71e4fa5df8a81d31"
server_key_iv: "a528d705ac4262f192d59dd42a85d5f3"
server_key_version: "encryptionkey@0"
I20260620 03:11:08.309027 22451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "7ce4594f2523402497677d37b5f8eca9"
format_stamp: "Formatted at 2026-06-20 03:11:08 on dist-test-slave-xv68"
server_key: "718b1e21dda634fc71e4fa5df8a81d31"
server_key_iv: "a528d705ac4262f192d59dd42a85d5f3"
server_key_version: "encryptionkey@0"
I20260620 03:11:08.313881 22451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 03:11:08.317307 22466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:11:08.318572 22451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:11:08.318702 22451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "7ce4594f2523402497677d37b5f8eca9"
format_stamp: "Formatted at 2026-06-20 03:11:08 on dist-test-slave-xv68"
server_key: "718b1e21dda634fc71e4fa5df8a81d31"
server_key_iv: "a528d705ac4262f192d59dd42a85d5f3"
server_key_version: "encryptionkey@0"
I20260620 03:11:08.318856 22451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:11:08.339169 22451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:11:08.343489 22451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:11:08.343716 22451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:11:08.361676 22451 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36741
I20260620 03:11:08.361716 22517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36741 every 8 connection(s)
I20260620 03:11:08.363027 22451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:11:08.366518 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 22451
I20260620 03:11:08.366674 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:11:08.366801 22518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:11:08.367175 388 external_mini_cluster.cc:1428] Setting key 5ba1340bf78c1ed65bced077d282371b
I20260620 03:11:08.381124 22518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9: Bootstrap starting.
Jun 20 03:11:08 dist-test-slave-xv68 krb5kdc[22438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:11:08.384941 22518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9: Neither blocks nor log segments found. Creating new log.
I20260620 03:11:08.386256 22518 log.cc:824] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9: Log is configured to *not* fsync() on all Append() calls
I20260620 03:11:08.389533 22518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9: No bootstrap required, opened a new log
I20260620 03:11:08.390733 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:08.368945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52811 (local address 127.0.97.62:36741)
0620 03:11:08.369811 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:08.369826 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:08.369876 (+ 50us) server_negotiation.cc:407] Connection header received
0620 03:11:08.371735 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:08.371762 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:08.372242 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:08.372679 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:08.373920 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:08.375904 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:08.377130 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:08.378210 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:08.378414 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:08.382370 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:08.382384 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:08.382407 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:08.382463 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:08.386417 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:08.387205 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:08.387211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:08.387216 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:08.387444 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:08.387839 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:08.387844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:08.387847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:08.388444 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:08.388714 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:08.389060 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 03:11:08.389449 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":360,"threads_started":1}
I20260620 03:11:08.394348 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:11:08.398201 22518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce4594f2523402497677d37b5f8eca9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36741 } }
I20260620 03:11:08.398748 22518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:11:08.398870 22518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce4594f2523402497677d37b5f8eca9, State: Initialized, Role: FOLLOWER
I20260620 03:11:08.399530 22518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce4594f2523402497677d37b5f8eca9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36741 } }
I20260620 03:11:08.399771 22518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:11:08.399915 22518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:11:08.400074 22518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:11:08.402179 22518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce4594f2523402497677d37b5f8eca9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36741 } }
I20260620 03:11:08.402674 22518 leader_election.cc:302] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ce4594f2523402497677d37b5f8eca9; no voters:
I20260620 03:11:08.403335 22518 leader_election.cc:288] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:11:08.403438 22528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:11:08.405913 22518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:11:08.410133 22528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9 [term 1 LEADER]: Becoming Leader. State: Replica: 7ce4594f2523402497677d37b5f8eca9, State: Running, Role: LEADER
I20260620 03:11:08.410812 22528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce4594f2523402497677d37b5f8eca9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36741 } }
I20260620 03:11:08.413245 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:08.398506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47005 (local address 127.0.97.62:36741)
0620 03:11:08.398694 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:08.398702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:08.399174 (+ 472us) server_negotiation.cc:407] Connection header received
0620 03:11:08.399386 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:08.399392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:08.399456 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:08.399563 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:08.401079 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:08.401799 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:08.403167 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:08.403950 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:08.404088 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:08.406661 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:08.406667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:08.406673 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:08.406708 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:08.409926 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:08.410666 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:08.410672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:08.410675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:08.410810 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:08.411247 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:08.411250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:08.411252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:08.411590 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:08.411862 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:08.412772 (+ 910us) server_negotiation.cc:299] Negotiation successful
0620 03:11:08.413017 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:11:08.420102 22530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ce4594f2523402497677d37b5f8eca9. Latest consensus state: current_term: 1 leader_uuid: "7ce4594f2523402497677d37b5f8eca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce4594f2523402497677d37b5f8eca9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36741 } } }
I20260620 03:11:08.420295 22529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ce4594f2523402497677d37b5f8eca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce4594f2523402497677d37b5f8eca9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36741 } } }
I20260620 03:11:08.420518 22530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9 [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:08.420518 22529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9 [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:08.425081 22542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:11:08.429911 22542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:11:08.438273 22542 catalog_manager.cc:1347] Generated new cluster ID: 26eb136b67794ba0b5f15dad344e4678
I20260620 03:11:08.438374 22542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:11:08.447703 22542 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:11:08.449432 22542 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:11:08.457334 22542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ce4594f2523402497677d37b5f8eca9: Generated new TSK 0
I20260620 03:11:08.458117 22542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:11:08.466781 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 22451
I20260620 03:11:08.468443 22513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:11:08.555932 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 22451
2026-06-20T03:11:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1978 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:11:08 dist-test-slave-xv68 krb5kdc[22550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:11:08 dist-test-slave-xv68 krb5kdc[22550](info): set up 2 sockets
Jun 20 03:11:08 dist-test-slave-xv68 krb5kdc[22550](info): commencing operation
krb5kdc: starting...
W20260620 03:11:09.659781 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:11:09 dist-test-slave-xv68 krb5kdc[22550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:11:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:11:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:11:09.900420 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40047
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:11:10.228747 22563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:11:10.229015 22563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:11:10.229106 22563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:11:10.236706 22563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:11:10.236815 22563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:11:10.236862 22563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:11:10.236928 22563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:11:10.236974 22563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:11:10.246592 22563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:11:10.247785 22563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:11:10.248996 22563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:11:10.260632 22571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:10.260609 22569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:10.260898 22568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:11:10.261752 22563 server_base.cc:1034] running on GCE node
I20260620 03:11:10.262436 22563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:11:10.263846 22563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:11:10.265156 22563 hybrid_clock.cc:648] HybridClock initialized: now 1781925070265108 us; error 39 us; skew 500 ppm
I20260620 03:11:10.265488 22563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:11:10 dist-test-slave-xv68 krb5kdc[22550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:11:10.268472 22563 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:11:10.269932 22563 webserver.cc:460] Webserver started at http://127.0.97.62:40719/ using document root <none> and password file <none>
I20260620 03:11:10.270586 22563 fs_manager.cc:359] Metadata directory not provided
I20260620 03:11:10.270759 22563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:11:10.271055 22563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:11:10.273960 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "0d752155c9f8428984042dc909c7b5cf"
format_stamp: "Formatted at 2026-06-20 03:11:10 on dist-test-slave-xv68"
server_key: "dcd947a11ddc4dd1198c8dc9ed0b2a74"
server_key_iv: "0ec966dcd39c01571c99096f8b08f6ff"
server_key_version: "encryptionkey@0"
I20260620 03:11:10.274720 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "0d752155c9f8428984042dc909c7b5cf"
format_stamp: "Formatted at 2026-06-20 03:11:10 on dist-test-slave-xv68"
server_key: "dcd947a11ddc4dd1198c8dc9ed0b2a74"
server_key_iv: "0ec966dcd39c01571c99096f8b08f6ff"
server_key_version: "encryptionkey@0"
I20260620 03:11:10.279713 22563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 03:11:10.282928 22578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:11:10.284252 22563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:11:10.284430 22563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "0d752155c9f8428984042dc909c7b5cf"
format_stamp: "Formatted at 2026-06-20 03:11:10 on dist-test-slave-xv68"
server_key: "dcd947a11ddc4dd1198c8dc9ed0b2a74"
server_key_iv: "0ec966dcd39c01571c99096f8b08f6ff"
server_key_version: "encryptionkey@0"
I20260620 03:11:10.284636 22563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:11:10.301280 22563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:11:10.305256 22563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:11:10.305538 22563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:11:10.323588 22563 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40047
I20260620 03:11:10.323720 22629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40047 every 8 connection(s)
I20260620 03:11:10.324892 22563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:11:10.328338 22630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:11:10.334019 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 22563
I20260620 03:11:10.334208 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:11:10.334614 388 external_mini_cluster.cc:1428] Setting key f6f36d8b37f667fb33a6a7e3c721005e
I20260620 03:11:10.338717 22630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf: Bootstrap starting.
I20260620 03:11:10.342140 22630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf: Neither blocks nor log segments found. Creating new log.
I20260620 03:11:10.343370 22630 log.cc:824] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf: Log is configured to *not* fsync() on all Append() calls
I20260620 03:11:10.346307 22630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf: No bootstrap required, opened a new log
I20260620 03:11:10.347471 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:10.336279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34199 (local address 127.0.97.62:40047)
0620 03:11:10.337083 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:10.337094 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:10.337136 (+ 42us) server_negotiation.cc:407] Connection header received
0620 03:11:10.338751 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:10.338771 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:10.339162 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:10.339514 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:10.340065 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:10.341356 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:10.343243 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:10.344148 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:10.344348 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:10.345352 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:10.345359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:10.345372 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:11:10.345872 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:10.345942 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:11:10.345965 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:10.346052 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:10.346302 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 03:11:10.346528 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":395,"threads_started":1}
I20260620 03:11:10.350837 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:11:10.353888 22630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d752155c9f8428984042dc909c7b5cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40047 } }
I20260620 03:11:10.354249 22630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:11:10.354359 22630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d752155c9f8428984042dc909c7b5cf, State: Initialized, Role: FOLLOWER
I20260620 03:11:10.354962 22630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d752155c9f8428984042dc909c7b5cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40047 } }
I20260620 03:11:10.355373 22630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:11:10.355618 22630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:11:10.355849 22630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:11:10.358218 22630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d752155c9f8428984042dc909c7b5cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40047 } }
I20260620 03:11:10.358690 22630 leader_election.cc:302] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d752155c9f8428984042dc909c7b5cf; no voters:
I20260620 03:11:10.359386 22630 leader_election.cc:288] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:11:10.359490 22640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:11:10.360872 22640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf [term 1 LEADER]: Becoming Leader. State: Replica: 0d752155c9f8428984042dc909c7b5cf, State: Running, Role: LEADER
I20260620 03:11:10.361759 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:10.355145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41553 (local address 127.0.97.62:40047)
0620 03:11:10.355351 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:10.355363 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:10.355794 (+ 431us) server_negotiation.cc:407] Connection header received
0620 03:11:10.355985 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:10.355990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:10.356054 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:10.356184 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:10.356650 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:10.357435 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:10.358865 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:10.359428 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:10.359563 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:10.360953 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:10.360958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:10.360961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:11:10.360976 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:10.361011 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:11:10.361025 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:10.361125 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:10.361296 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 03:11:10.361397 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 03:11:10.362089 22630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:11:10.362594 22640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d752155c9f8428984042dc909c7b5cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40047 } }
I20260620 03:11:10.365550 22642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d752155c9f8428984042dc909c7b5cf. Latest consensus state: current_term: 1 leader_uuid: "0d752155c9f8428984042dc909c7b5cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d752155c9f8428984042dc909c7b5cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40047 } } }
I20260620 03:11:10.365648 22641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d752155c9f8428984042dc909c7b5cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d752155c9f8428984042dc909c7b5cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40047 } } }
I20260620 03:11:10.366051 22641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:10.365952 22642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:10.367643 22648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:11:10.374081 22648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:11:10.382134 22648 catalog_manager.cc:1347] Generated new cluster ID: 2aba1d0792534965a2d9368ab7b2bb4a
I20260620 03:11:10.382246 22648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:11:10.405751 22648 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:11:10.407618 22648 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:11:10.422123 22648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d752155c9f8428984042dc909c7b5cf: Generated new TSK 0
I20260620 03:11:10.423163 22648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:11:10.440457 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 22563
W20260620 03:11:10.441555 22625 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34199
W20260620 03:11:10.442270 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 22563: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:11:10.442415 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 22563
2026-06-20T03:11:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1886 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:11:10 dist-test-slave-xv68 krb5kdc[22661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:11:10 dist-test-slave-xv68 krb5kdc[22661](info): set up 2 sockets
Jun 20 03:11:10 dist-test-slave-xv68 krb5kdc[22661](info): commencing operation
krb5kdc: starting...
W20260620 03:11:11.547639 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:11:11 dist-test-slave-xv68 krb5kdc[22661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:11:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:11:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:11:11.782939 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38637
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:11:12.101581 22674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:11:12.101861 22674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:11:12.102016 22674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:11:12.109030 22674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:11:12.109126 22674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:11:12.109226 22674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:11:12.109324 22674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:11:12.109407 22674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:11:12.118817 22674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:11:12.120059 22674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:11:12.121280 22674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:11:12.133257 22682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:12.133323 22679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:12.133350 22680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:11:12.134397 22674 server_base.cc:1034] running on GCE node
I20260620 03:11:12.135113 22674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:11:12.136459 22674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:11:12.137698 22674 hybrid_clock.cc:648] HybridClock initialized: now 1781925072137658 us; error 36 us; skew 500 ppm
I20260620 03:11:12.138024 22674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:11:12 dist-test-slave-xv68 krb5kdc[22661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:11:12.144971 22674 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:11:12.146400 22674 webserver.cc:460] Webserver started at http://127.0.97.62:33829/ using document root <none> and password file <none>
I20260620 03:11:12.147039 22674 fs_manager.cc:359] Metadata directory not provided
I20260620 03:11:12.147187 22674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:11:12.147532 22674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:11:12.150274 22674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "7b661a41a7914147bdf0481cc464f953"
format_stamp: "Formatted at 2026-06-20 03:11:12 on dist-test-slave-xv68"
server_key: "8d13309b386e6d645f3b66e31a5e2a4e"
server_key_iv: "dda3455acf990d97abaf266ddfed9d62"
server_key_version: "encryptionkey@0"
I20260620 03:11:12.150987 22674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "7b661a41a7914147bdf0481cc464f953"
format_stamp: "Formatted at 2026-06-20 03:11:12 on dist-test-slave-xv68"
server_key: "8d13309b386e6d645f3b66e31a5e2a4e"
server_key_iv: "dda3455acf990d97abaf266ddfed9d62"
server_key_version: "encryptionkey@0"
I20260620 03:11:12.156168 22674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 03:11:12.159497 22689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:11:12.160750 22674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:11:12.160930 22674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "7b661a41a7914147bdf0481cc464f953"
format_stamp: "Formatted at 2026-06-20 03:11:12 on dist-test-slave-xv68"
server_key: "8d13309b386e6d645f3b66e31a5e2a4e"
server_key_iv: "dda3455acf990d97abaf266ddfed9d62"
server_key_version: "encryptionkey@0"
I20260620 03:11:12.161127 22674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:11:12.166661 22674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:11:12.170575 22674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:11:12.170873 22674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:11:12.189337 22674 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38637
I20260620 03:11:12.189383 22740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38637 every 8 connection(s)
I20260620 03:11:12.190765 22674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:11:12.193936 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 22674
I20260620 03:11:12.194123 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:11:12.194476 22741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:11:12.194578 388 external_mini_cluster.cc:1428] Setting key a7391ab11244474e75114cc930740064
I20260620 03:11:12.202395 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:12.196254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44171 (local address 127.0.97.62:38637)
0620 03:11:12.197024 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:12.197035 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:12.197077 (+ 42us) server_negotiation.cc:407] Connection header received
0620 03:11:12.198930 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:12.198956 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:12.199270 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:12.199649 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:12.200842 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:12.200848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:12.200859 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:11:12.200873 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:12.200927 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:11:12.200948 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:12.201112 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:12.201282 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 03:11:12.201482 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":362,"threads_started":1}
I20260620 03:11:12.206105 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:11:12.209031 22741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953: Bootstrap starting.
I20260620 03:11:12.212663 22741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953: Neither blocks nor log segments found. Creating new log.
I20260620 03:11:12.212687 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:12.210359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42813 (local address 127.0.97.62:38637)
0620 03:11:12.210559 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:12.210565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:12.210880 (+ 315us) server_negotiation.cc:407] Connection header received
0620 03:11:12.211038 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:12.211041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:12.211084 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:12.211161 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:12.212040 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:12.212044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:12.212047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:11:12.212061 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:12.212096 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:11:12.212110 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:12.212198 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:12.212333 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 03:11:12.212375 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 03:11:12.213754 22741 log.cc:824] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953: Log is configured to *not* fsync() on all Append() calls
I20260620 03:11:12.216228 22741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953: No bootstrap required, opened a new log
I20260620 03:11:12.222982 22741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b661a41a7914147bdf0481cc464f953" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38637 } }
I20260620 03:11:12.223407 22741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:11:12.223516 22741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b661a41a7914147bdf0481cc464f953, State: Initialized, Role: FOLLOWER
I20260620 03:11:12.224050 22741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b661a41a7914147bdf0481cc464f953" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38637 } }
I20260620 03:11:12.224291 22741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:11:12.224432 22741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:11:12.224596 22741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:11:12.226632 22741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b661a41a7914147bdf0481cc464f953" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38637 } }
I20260620 03:11:12.227097 22741 leader_election.cc:302] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b661a41a7914147bdf0481cc464f953; no voters:
I20260620 03:11:12.227682 22741 leader_election.cc:288] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:11:12.227828 22751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:11:12.228489 22751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953 [term 1 LEADER]: Becoming Leader. State: Replica: 7b661a41a7914147bdf0481cc464f953, State: Running, Role: LEADER
I20260620 03:11:12.229048 22751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b661a41a7914147bdf0481cc464f953" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38637 } }
I20260620 03:11:12.229619 22741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:11:12.232033 22752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b661a41a7914147bdf0481cc464f953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b661a41a7914147bdf0481cc464f953" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38637 } } }
I20260620 03:11:12.232429 22752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953 [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:12.232064 22753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b661a41a7914147bdf0481cc464f953. Latest consensus state: current_term: 1 leader_uuid: "7b661a41a7914147bdf0481cc464f953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b661a41a7914147bdf0481cc464f953" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38637 } } }
I20260620 03:11:12.232863 22753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953 [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:12.234767 22758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:11:12.239822 22758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:11:12.248559 22758 catalog_manager.cc:1347] Generated new cluster ID: 00df4b39c35a441e99dc3fbbbf931e7b
I20260620 03:11:12.248673 22758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:11:12.261873 22758 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:11:12.262468 22758 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:11:12.272010 22758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b661a41a7914147bdf0481cc464f953: Generated new TSK 0
I20260620 03:11:12.272710 22758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:11:12.278321 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 22674
W20260620 03:11:12.279397 22736 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44171
W20260620 03:11:12.280027 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 22674: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:11:12.280169 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 22674
2026-06-20T03:11:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1837 ms)
[----------] 18 tests from AuthTokenIssuingTest (34319 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:11:12 dist-test-slave-xv68 krb5kdc[22772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:11:12 dist-test-slave-xv68 krb5kdc[22772](info): set up 2 sockets
Jun 20 03:11:12 dist-test-slave-xv68 krb5kdc[22772](info): commencing operation
krb5kdc: starting...
W20260620 03:11:13.391898 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:11:13 dist-test-slave-xv68 krb5kdc[22772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:11:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:11:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:11:13.643849 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43615
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43615,127.0.97.61:33055,127.0.97.60:41719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:11:13.965191 22785 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 03:11:13.965500 22785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:11:13.965566 22785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:11:13.965649 22785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:11:13.972947 22785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:11:13.973029 22785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:11:13.973088 22785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:11:13.973136 22785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:11:13.973203 22785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:11:13.982834 22785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43615,127.0.97.61:33055,127.0.97.60:41719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:11:13.984054 22785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:11:13.985289 22785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:11:13.996657 22791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:13.996714 22793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:13.996686 22790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:11:13.997795 22785 server_base.cc:1034] running on GCE node
I20260620 03:11:13.998615 22785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:11:14.000025 22785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:11:14.001287 22785 hybrid_clock.cc:648] HybridClock initialized: now 1781925074001212 us; error 52 us; skew 500 ppm
I20260620 03:11:14.001629 22785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:11:13 dist-test-slave-xv68 krb5kdc[22772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:11:14.004945 22785 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:11:14.006460 22785 webserver.cc:460] Webserver started at http://127.0.97.62:33143/ using document root <none> and password file <none>
I20260620 03:11:14.007090 22785 fs_manager.cc:359] Metadata directory not provided
I20260620 03:11:14.007189 22785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:11:14.007529 22785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:11:14.010342 22785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0"
format_stamp: "Formatted at 2026-06-20 03:11:14 on dist-test-slave-xv68"
server_key: "830c0efbc8a582345dff7294a0ba7975"
server_key_iv: "fcd949cfe0b205ff188122e9e2a76d93"
server_key_version: "encryptionkey@0"
I20260620 03:11:14.011044 22785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0"
format_stamp: "Formatted at 2026-06-20 03:11:14 on dist-test-slave-xv68"
server_key: "830c0efbc8a582345dff7294a0ba7975"
server_key_iv: "fcd949cfe0b205ff188122e9e2a76d93"
server_key_version: "encryptionkey@0"
I20260620 03:11:14.016144 22785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 03:11:14.019470 22800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:11:14.020743 22785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:11:14.020871 22785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0"
format_stamp: "Formatted at 2026-06-20 03:11:14 on dist-test-slave-xv68"
server_key: "830c0efbc8a582345dff7294a0ba7975"
server_key_iv: "fcd949cfe0b205ff188122e9e2a76d93"
server_key_version: "encryptionkey@0"
I20260620 03:11:14.021044 22785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:11:14.061069 22785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:11:14.065012 22785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:11:14.065256 22785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:11:14.083595 22785 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43615
I20260620 03:11:14.083614 22851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43615 every 8 connection(s)
I20260620 03:11:14.084954 22785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:11:14.086380 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 22785
I20260620 03:11:14.086593 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:11:14.087446 388 external_mini_cluster.cc:1428] Setting key a92624d1e28fa81e77d558be8a90535f
I20260620 03:11:14.089498 22852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:11:14.099084 22852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Jun 20 03:11:14 dist-test-slave-xv68 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 03:11:14.121575 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.102229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43615 (local address 127.0.0.1:58988)
0620 03:11:14.103100 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.103150 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 03:11:14.105127 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:11:14.105951 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:11:14.105957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:11:14.106656 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:11:14.106977 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:14.106993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:14.108610 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:14.108615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:14.109792 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:14.109804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:14.110705 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:14.110713 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:14.110881 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:14.111561 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:11:14.111594 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:11:14.113931 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:11:14.117888 (+ 3957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:14.117898 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:14.117906 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:14.118263 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:14.118736 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:14.118739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:14.118741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:14.118889 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:14.119826 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:11:14.119832 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:11:14.120018 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 03:11:14.120223 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 03:11:14.120587 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":404,"threads_started":1}
I20260620 03:11:14.121618 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.102248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.0.97.62:43615)
0620 03:11:14.103945 (+ 1697us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:14.103953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:14.104000 (+ 47us) server_negotiation.cc:407] Connection header received
0620 03:11:14.105513 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:14.105518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:14.105738 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:14.105944 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:14.107196 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:14.108455 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:14.109930 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:14.110470 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:14.110686 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:14.114151 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:14.114159 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:14.114169 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:14.114205 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:14.117744 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:14.118399 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:14.118411 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:14.118418 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:14.118616 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:14.118999 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:14.119007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:14.119013 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:14.119693 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:14.119897 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:14.120405 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 03:11:14.120799 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":1483,"thread_start_us":1250,"threads_started":1}
I20260620 03:11:14.124437 22852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:11:14.125557 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.125159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33055 (local address 127.0.0.1:48040)
0620 03:11:14.125347 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.125458 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 03:11:14.125857 22803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33055: connect: Connection refused (error 111)
I20260620 03:11:14.127523 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.127079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33055 (local address 127.0.0.1:48046)
0620 03:11:14.127257 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.127366 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 03:11:14.127748 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33055: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33055: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 03:11:14.146776 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33055
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43615,127.0.97.61:33055,127.0.97.60:41719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:11:14.177112 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } attempt: 1
I20260620 03:11:14.178364 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.177840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33055 (local address 127.0.0.1:48060)
0620 03:11:14.178071 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.178183 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 03:11:14.179397 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.178986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33055 (local address 127.0.0.1:48066)
0620 03:11:14.179138 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.179219 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 03:11:14.179608 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33055: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33055: connect: Connection refused (error 111)
I20260620 03:11:14.247900 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } attempt: 2
I20260620 03:11:14.249238 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.248647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33055 (local address 127.0.0.1:48076)
0620 03:11:14.248928 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.249063 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 03:11:14.250433 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.249944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33055 (local address 127.0.0.1:48092)
0620 03:11:14.250140 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.250239 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 03:11:14.250661 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33055: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33055: connect: Connection refused (error 111)
I20260620 03:11:14.341944 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } attempt: 3
I20260620 03:11:14.343214 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.342688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33055 (local address 127.0.0.1:48098)
0620 03:11:14.342926 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.343037 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 03:11:14.344337 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.343920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33055 (local address 127.0.0.1:48114)
0620 03:11:14.344088 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.344183 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 03:11:14.344550 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33055: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33055: connect: Connection refused (error 111)
W20260620 03:11:14.471747 22859 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 03:11:14.472045 22859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:11:14.472110 22859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:11:14.472193 22859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:11:14.479424 22859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:11:14.479512 22859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:11:14.479569 22859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:11:14.479635 22859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:11:14.479692 22859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:11:14.487829 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } attempt: 4
I20260620 03:11:14.489161 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.488591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33055 (local address 127.0.0.1:48120)
0620 03:11:14.488883 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.489028 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260620 03:11:14.489388 22859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43615,127.0.97.61:33055,127.0.97.60:41719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:11:14.490255 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.489822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33055 (local address 127.0.0.1:48134)
0620 03:11:14.490006 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.490106 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 03:11:14.490499 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33055: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33055: connect: Connection refused (error 111)
I20260620 03:11:14.490934 22859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:11:14.492254 22859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:11:14.503551 22864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:14.503607 22867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:14.503912 22865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:11:14.504779 22859 server_base.cc:1034] running on GCE node
I20260620 03:11:14.505569 22859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:11:14.506917 22859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:11:14.508162 22859 hybrid_clock.cc:648] HybridClock initialized: now 1781925074508094 us; error 42 us; skew 500 ppm
I20260620 03:11:14.508508 22859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:11:14 dist-test-slave-xv68 krb5kdc[22772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:11:14.511705 22859 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 03:11:14.513033 22859 webserver.cc:460] Webserver started at http://127.0.97.61:34583/ using document root <none> and password file <none>
I20260620 03:11:14.513724 22859 fs_manager.cc:359] Metadata directory not provided
I20260620 03:11:14.513852 22859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:11:14.514161 22859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:11:14.517207 22859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-1/data/instance:
uuid: "bd2d848849ea40de8642a880d98295a9"
format_stamp: "Formatted at 2026-06-20 03:11:14 on dist-test-slave-xv68"
server_key: "b94299750a7ed38689a20a800fcca870"
server_key_iv: "593b416d96da0009c0035dbe4bee388b"
server_key_version: "encryptionkey@0"
I20260620 03:11:14.517990 22859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-1/wal/instance:
uuid: "bd2d848849ea40de8642a880d98295a9"
format_stamp: "Formatted at 2026-06-20 03:11:14 on dist-test-slave-xv68"
server_key: "b94299750a7ed38689a20a800fcca870"
server_key_iv: "593b416d96da0009c0035dbe4bee388b"
server_key_version: "encryptionkey@0"
I20260620 03:11:14.522946 22859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 03:11:14.526551 22874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:11:14.527938 22859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:11:14.528118 22859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-1/wal
uuid: "bd2d848849ea40de8642a880d98295a9"
format_stamp: "Formatted at 2026-06-20 03:11:14 on dist-test-slave-xv68"
server_key: "b94299750a7ed38689a20a800fcca870"
server_key_iv: "593b416d96da0009c0035dbe4bee388b"
server_key_version: "encryptionkey@0"
I20260620 03:11:14.528337 22859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:11:14.548998 22859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:11:14.553474 22859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:11:14.553766 22859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:11:14.572275 22859 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:33055
I20260620 03:11:14.572299 22925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:33055 every 8 connection(s)
I20260620 03:11:14.573645 22859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-1/data/info.pb
I20260620 03:11:14.577212 22926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:11:14.580116 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 22859
I20260620 03:11:14.580338 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-1/wal/instance
I20260620 03:11:14.580780 388 external_mini_cluster.cc:1428] Setting key 9368b35f2054f9aca38820aa25e6825a
I20260620 03:11:14.583758 22926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:11:14 dist-test-slave-xv68 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260620 03:11:14.605937 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.586217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59002 (local address 127.0.97.62:43615)
0620 03:11:14.586392 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:14.586396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:14.587715 (+ 1319us) server_negotiation.cc:407] Connection header received
0620 03:11:14.589563 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:14.589578 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:14.589667 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:14.589780 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:14.591177 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:14.592025 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:14.594415 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:14.594875 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:14.594980 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:14.599126 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:14.599135 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:14.599143 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:14.599208 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:14.602489 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:14.603487 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:14.603491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:14.603493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:14.603610 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:14.604148 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:14.604152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:14.604153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:14.604428 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:14.604546 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:14.605478 (+ 932us) server_negotiation.cc:299] Negotiation successful
0620 03:11:14.605643 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 03:11:14.607102 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.586772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43615 (local address 127.0.0.1:59002)
0620 03:11:14.587551 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.587595 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 03:11:14.589285 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:11:14.590002 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:11:14.590009 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:11:14.590484 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:11:14.591000 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:14.591016 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:14.592215 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:14.592224 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:14.594245 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:14.594262 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:14.595103 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:14.595112 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:14.595468 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:14.596171 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:11:14.596210 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:11:14.598891 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:11:14.602650 (+ 3759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:14.602662 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:14.602682 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:14.603371 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:14.603732 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:14.603738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:14.603744 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:14.604015 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:14.604565 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:11:14.604573 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:11:14.605062 (+ 489us) client_negotiation.cc:769] Sending connection context
0620 03:11:14.605400 (+ 338us) client_negotiation.cc:240] Negotiation successful
0620 03:11:14.605944 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":325,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260620 03:11:14.609112 22926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:11:14 dist-test-slave-xv68 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:11:14.624063 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.609966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33055 (local address 127.0.0.1:48138)
0620 03:11:14.610192 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.610217 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 03:11:14.610345 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:11:14.611135 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:11:14.611138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:11:14.611531 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:11:14.611664 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:14.611673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:14.613599 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:14.613602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:14.614420 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:14.614430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:14.615240 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:14.615246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:14.615451 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:14.616029 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:11:14.616045 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:11:14.618451 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:11:14.621607 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:14.621621 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:14.621625 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:14.621965 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:14.622328 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:14.622335 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:14.622338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:14.622473 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:14.623048 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:11:14.623052 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:11:14.623201 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 03:11:14.623396 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 03:11:14.623689 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 03:11:14.624184 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.609979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48138 (local address 127.0.97.61:33055)
0620 03:11:14.610629 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:14.610640 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:14.610684 (+ 44us) server_negotiation.cc:407] Connection header received
0620 03:11:14.610739 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:14.610749 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:14.610968 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:14.611110 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:14.611839 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:14.613450 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:14.614553 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:14.615079 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:14.615224 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:14.618658 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:14.618668 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:14.618680 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:14.618707 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:14.621466 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:14.622094 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:14.622102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:14.622105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:14.622225 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:14.622581 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:14.622586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:14.622590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:14.622904 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:14.623135 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:14.623541 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 03:11:14.623801 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":452,"thread_start_us":280,"threads_started":1}
I20260620 03:11:14.626586 22926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:11:14.627810 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.627368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39286)
0620 03:11:14.627545 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.627670 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 03:11:14.628207 22876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
I20260620 03:11:14.629776 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.629469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39292)
0620 03:11:14.629603 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.629681 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 03:11:14.630096 22926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41719: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 03:11:14.640012 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41719
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43615,127.0.97.61:33055,127.0.97.60:41719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:11:14.679431 22926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } attempt: 1
I20260620 03:11:14.680717 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.680239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39296)
0620 03:11:14.680444 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.680544 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 03:11:14.681717 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.681364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39298)
0620 03:11:14.681511 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.681607 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 03:11:14.681926 22926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41719: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
I20260620 03:11:14.750144 22926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } attempt: 2
I20260620 03:11:14.751418 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.750901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39312)
0620 03:11:14.751137 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.751228 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 03:11:14.752416 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.752075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39322)
0620 03:11:14.752232 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.752300 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 03:11:14.752665 22926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41719: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
I20260620 03:11:14.789765 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } attempt: 5
Jun 20 03:11:14 dist-test-slave-xv68 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:11:14.803650 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.790724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48144 (local address 127.0.97.61:33055)
0620 03:11:14.790941 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:14.790946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:14.790969 (+ 23us) server_negotiation.cc:407] Connection header received
0620 03:11:14.791155 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:14.791166 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:14.791232 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:14.791380 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:14.792201 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:14.792894 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:14.794329 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:14.794809 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:14.794950 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:14.798111 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:14.798118 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:14.798123 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:14.798166 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:14.801040 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:14.801725 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:14.801737 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:14.801740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:14.801856 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:14.802227 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:14.802233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:14.802237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:14.802512 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:14.802670 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:14.803063 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 03:11:14.803284 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 03:11:14.803650 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.790591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33055 (local address 127.0.0.1:48144)
0620 03:11:14.790830 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.790854 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 03:11:14.790989 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:11:14.791410 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:11:14.791414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:11:14.791842 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:11:14.792009 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:14.792017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:14.793042 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:14.793048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:14.794151 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:14.794165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:14.794973 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:14.794978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:14.795168 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:14.795833 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:11:14.795853 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:11:14.797883 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:11:14.801189 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:14.801195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:14.801200 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:14.801595 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:14.801967 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:14.801973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:14.801976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:14.802119 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:14.802638 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:11:14.802643 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:11:14.802811 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 03:11:14.802956 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 03:11:14.803248 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 03:11:14.804916 22852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:11:14.806002 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.805531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39334)
0620 03:11:14.805727 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.805819 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 03:11:14.806979 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.806617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39338)
0620 03:11:14.806766 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.806860 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 03:11:14.807180 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41719: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
I20260620 03:11:14.843917 22926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } attempt: 3
I20260620 03:11:14.845196 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.844615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39340)
0620 03:11:14.844870 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.844983 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 03:11:14.846436 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.845869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39352)
0620 03:11:14.846137 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.846249 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260620 03:11:14.846694 22926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41719: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
I20260620 03:11:14.858408 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } attempt: 1
I20260620 03:11:14.859589 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.859079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39368)
0620 03:11:14.859277 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.859417 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 03:11:14.860743 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.860375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39384)
0620 03:11:14.860543 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.860629 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 03:11:14.860984 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41719: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
I20260620 03:11:14.929225 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } attempt: 2
I20260620 03:11:14.930356 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.929884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39386)
0620 03:11:14.930110 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.930194 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 03:11:14.931447 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.931076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39390)
0620 03:11:14.931221 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.931281 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 03:11:14.931699 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41719: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
W20260620 03:11:14.971907 22933 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 03:11:14.972285 22933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:11:14.972404 22933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:11:14.972532 22933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:11:14.979839 22933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:11:14.979962 22933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:11:14.980082 22933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:11:14.980163 22933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:11:14.980250 22933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:11:14.989936 22926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } attempt: 4
I20260620 03:11:14.989985 22933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43615,127.0.97.61:33055,127.0.97.60:41719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:11:14.991088 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.990586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39398)
0620 03:11:14.990832 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.990932 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 03:11:14.991472 22933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:11:14.992169 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:14.991864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39400)
0620 03:11:14.991990 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:14.992055 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 03:11:14.992408 22926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41719: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
I20260620 03:11:14.993031 22933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:11:15.004709 22939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:15.004740 22941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:15.004750 22938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:11:15.008826 22933 server_base.cc:1034] running on GCE node
I20260620 03:11:15.009583 22933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:11:15.010932 22933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:11:15.012166 22933 hybrid_clock.cc:648] HybridClock initialized: now 1781925075012099 us; error 44 us; skew 500 ppm
I20260620 03:11:15.012492 22933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:11:15 dist-test-slave-xv68 krb5kdc[22772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:11:15.015614 22933 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 03:11:15.017117 22933 webserver.cc:460] Webserver started at http://127.0.97.60:45455/ using document root <none> and password file <none>
I20260620 03:11:15.017814 22933 fs_manager.cc:359] Metadata directory not provided
I20260620 03:11:15.017937 22933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:11:15.018239 22933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:11:15.021162 22933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-2/data/instance:
uuid: "14753a319fe446aead99bf1927d7b323"
format_stamp: "Formatted at 2026-06-20 03:11:15 on dist-test-slave-xv68"
server_key: "8445384e5edd8ee74e1838344e8575e6"
server_key_iv: "4133a3a8d4a15cf645fd5c08ba8cd6f9"
server_key_version: "encryptionkey@0"
I20260620 03:11:15.021937 22933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-2/wal/instance:
uuid: "14753a319fe446aead99bf1927d7b323"
format_stamp: "Formatted at 2026-06-20 03:11:15 on dist-test-slave-xv68"
server_key: "8445384e5edd8ee74e1838344e8575e6"
server_key_iv: "4133a3a8d4a15cf645fd5c08ba8cd6f9"
server_key_version: "encryptionkey@0"
I20260620 03:11:15.027025 22933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 03:11:15.030241 22948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:11:15.031567 22933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:11:15.031746 22933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-2/wal
uuid: "14753a319fe446aead99bf1927d7b323"
format_stamp: "Formatted at 2026-06-20 03:11:15 on dist-test-slave-xv68"
server_key: "8445384e5edd8ee74e1838344e8575e6"
server_key_iv: "4133a3a8d4a15cf645fd5c08ba8cd6f9"
server_key_version: "encryptionkey@0"
I20260620 03:11:15.031947 22933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:11:15.037923 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } attempt: 3
I20260620 03:11:15.039106 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.038496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39416)
0620 03:11:15.038794 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:15.038904 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260620 03:11:15.040124 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.039761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39420)
0620 03:11:15.039956 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:15.040016 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260620 03:11:15.040313 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41719: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41719: connect: Connection refused (error 111)
I20260620 03:11:15.054224 22933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:11:15.058640 22933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:11:15.058908 22933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:11:15.075812 22933 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41719
I20260620 03:11:15.075845 22999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41719 every 8 connection(s)
I20260620 03:11:15.077033 22933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-2/data/info.pb
I20260620 03:11:15.080554 23000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:11:15.083035 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 22933
I20260620 03:11:15.083218 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924720365844-388-0/minicluster-data/master-2/wal/instance
I20260620 03:11:15.083777 388 external_mini_cluster.cc:1428] Setting key ae6f126474f7a4cd6432121e64af5fcc
I20260620 03:11:15.090196 23000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:11:15 dist-test-slave-xv68 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 03:11:15 dist-test-slave-xv68 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 03:11:15.115564 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.094477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59018 (local address 127.0.97.62:43615)
0620 03:11:15.094737 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.094746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.098313 (+ 3567us) server_negotiation.cc:407] Connection header received
0620 03:11:15.100237 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.100242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.100323 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.100447 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.102024 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.103168 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.105045 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.105502 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.105629 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.109461 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:15.109466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:15.109470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:15.109502 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:15.112336 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.113133 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.113137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.113139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.113261 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.113708 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.113711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.113713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.113959 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:15.114067 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:15.114945 (+ 878us) server_negotiation.cc:299] Negotiation successful
0620 03:11:15.115166 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 03:11:15.116593 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.093892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43615 (local address 127.0.0.1:59018)
0620 03:11:15.098166 (+ 4274us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:15.098205 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 03:11:15.099996 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:11:15.100531 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:11:15.100538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:11:15.101339 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:11:15.101827 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:15.101850 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.103388 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.103393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:15.104884 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:15.104903 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.105713 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.105717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:15.105896 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.106612 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:11:15.106648 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:11:15.109266 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:11:15.112451 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:15.112460 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:15.112474 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:15.113021 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:15.113358 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:15.113363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:15.113366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:15.113593 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:15.114078 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:11:15.114086 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:11:15.114536 (+ 450us) client_negotiation.cc:769] Sending connection context
0620 03:11:15.114831 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 03:11:15.115215 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4050,"thread_start_us":3924,"threads_started":1}
I20260620 03:11:15.116619 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.085596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39434 (local address 127.0.97.60:41719)
0620 03:11:15.097397 (+ 11801us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.097415 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.098479 (+ 1064us) server_negotiation.cc:407] Connection header received
0620 03:11:15.100024 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.100053 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.100342 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.100535 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.102303 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.103903 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.105514 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.106212 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.106338 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.110461 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:15.110469 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:15.110480 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:15.110511 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:15.113745 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.114447 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.114454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.114456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.114576 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.114977 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.114981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.114984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.115440 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:15.115770 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:15.116019 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 03:11:15.116216 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11533,"thread_start_us":11737,"threads_started":1}
I20260620 03:11:15.117977 23000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:11:15 dist-test-slave-xv68 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 03:11:15 dist-test-slave-xv68 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:11:15.131635 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.118610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33055 (local address 127.0.0.1:48156)
0620 03:11:15.118791 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:15.118810 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:11:15.118909 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:11:15.119222 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:11:15.119225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:11:15.119695 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:11:15.119837 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:15.119844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.120926 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.120933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:15.121949 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:15.121960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.122712 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.122719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:15.122889 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.123406 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:11:15.123428 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:11:15.125871 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:11:15.129320 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:15.129327 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:15.129332 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:15.129655 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:15.130052 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:15.130058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:15.130063 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:15.130251 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:15.130882 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:11:15.130885 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:11:15.131007 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 03:11:15.131119 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 03:11:15.131333 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 03:11:15.132125 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.118698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.0.97.61:33055)
0620 03:11:15.118903 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.118910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.118928 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:11:15.119018 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.119025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.119104 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.119207 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.119973 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.120775 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.122081 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.122575 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.122665 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.126092 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:15.126097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:15.126101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:15.126133 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:15.129189 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.129763 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.129769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.129773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.129930 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.130355 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.130362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.130366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.130769 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:15.130928 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:15.131248 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 03:11:15.131502 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:11:15.133070 23000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:11:15.135149 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.119629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.0.97.62:43615)
0620 03:11:15.119775 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.119781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.119817 (+ 36us) server_negotiation.cc:407] Connection header received
0620 03:11:15.120024 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.120031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.120100 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.120190 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.121384 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.122147 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.123692 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.124056 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.124146 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.128479 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:15.128485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:15.128489 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:15.128531 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:15.131800 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.132663 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.132668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.132670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.132828 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.133294 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.133298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.133300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.133871 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:15.134115 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:15.134557 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 03:11:15.134767 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 03:11:15 dist-test-slave-xv68 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 03:11:15 dist-test-slave-xv68 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:11:15.148128 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.133829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39442)
0620 03:11:15.133958 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:15.133974 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:11:15.134076 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:11:15.134400 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:11:15.134404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:11:15.134812 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:11:15.134962 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:15.134973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.136168 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.136173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:15.137365 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:15.137375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.138339 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.138343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:15.138549 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.139057 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:11:15.139079 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:11:15.141308 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:11:15.145473 (+ 4165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:15.145478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:15.145483 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:15.146029 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:15.146407 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:15.146414 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:15.146421 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:15.146628 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:15.147153 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:11:15.147158 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:11:15.147344 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 03:11:15.147494 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 03:11:15.147782 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 03:11:15.148434 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.133770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39442 (local address 127.0.97.60:41719)
0620 03:11:15.133932 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.133938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.134017 (+ 79us) server_negotiation.cc:407] Connection header received
0620 03:11:15.134183 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.134188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.134256 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.134348 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.135124 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.135928 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.137492 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.138209 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.138334 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.141498 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:15.141504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:15.141507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:15.141578 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:15.145322 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.146171 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.146177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.146180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.146298 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.146755 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.146759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.146761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.147021 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:15.147153 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:15.147616 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 03:11:15.148031 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":36}
I20260620 03:11:15.152957 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.136680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.0.97.61:33055)
0620 03:11:15.136832 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.136837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.136854 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:11:15.137024 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.137030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.137100 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.137194 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.138554 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.139702 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.140995 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.141575 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.141669 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.147135 (+ 5466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:15.147141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:15.147146 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:15.147183 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:15.150430 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.151074 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.151084 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.151086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.151216 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.151640 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.151648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.151651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.151894 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:15.152170 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:15.152430 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 03:11:15.152664 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 03:11:15.153340 23000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323: Bootstrap starting.
I20260620 03:11:15.156765 23000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323: Neither blocks nor log segments found. Creating new log.
I20260620 03:11:15.157362 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:11:15.157804 23000 log.cc:824] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323: Log is configured to *not* fsync() on all Append() calls
I20260620 03:11:15.160400 23000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323: No bootstrap required, opened a new log
I20260620 03:11:15.159919 22827 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59032
I20260620 03:11:15.167258 23000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } } peers { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } } peers { permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } }
I20260620 03:11:15.167667 23000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14753a319fe446aead99bf1927d7b323, State: Initialized, Role: FOLLOWER
I20260620 03:11:15.168078 23000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } } peers { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } } peers { permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } }
I20260620 03:11:15.169055 23006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } } peers { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } } peers { permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } } }
I20260620 03:11:15.169443 23006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:11:15.170063 23000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 03:11:15.179272 23017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 03:11:15.179426 23017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 03:11:15.177112 22933 master_runner.cc:186] Error getting master registration for 127.0.97.62:43615: OK, instance_id { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" instance_seqno: 1781925074078636 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 03:11:15.186122 22933 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 03:11:15.217525 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } attempt: 4
Jun 20 03:11:15 dist-test-slave-xv68 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 03:11:15.233974 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.218394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39458 (local address 127.0.97.60:41719)
0620 03:11:15.218604 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.218612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.218641 (+ 29us) server_negotiation.cc:407] Connection header received
0620 03:11:15.218852 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.218857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.218938 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.219049 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.220069 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.221218 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.222989 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.223703 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.223817 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.227485 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:15.227489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:15.227492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:15.227523 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:15.231001 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.231712 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.231717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.231721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.231879 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.232292 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.232297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.232300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.232722 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:15.232882 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:15.233372 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 03:11:15.233584 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 03:11:15.233968 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.218267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39458)
0620 03:11:15.218499 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:15.218533 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 03:11:15.218689 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:11:15.219101 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:11:15.219106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:11:15.219682 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:11:15.219920 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:15.219934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.221387 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.221393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:15.222851 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:15.222866 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.224068 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.224073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:15.224199 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.224845 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:11:15.224871 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:11:15.227263 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:11:15.231135 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:15.231141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:15.231145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:15.231587 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:15.231995 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:15.232001 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:15.232005 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:15.232166 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:15.232940 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:11:15.232944 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:11:15.233111 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 03:11:15.233249 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 03:11:15.233570 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 03:11:15.240140 22852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0: Bootstrap starting.
I20260620 03:11:15.244217 22852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0: Neither blocks nor log segments found. Creating new log.
I20260620 03:11:15.245507 22852 log.cc:824] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0: Log is configured to *not* fsync() on all Append() calls
I20260620 03:11:15.248831 22852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0: No bootstrap required, opened a new log
I20260620 03:11:15.256448 22852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } } peers { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } } peers { permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } }
I20260620 03:11:15.256848 22852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a4ea2f4b1d5426c8f05bfc2dcc35bd0, State: Initialized, Role: FOLLOWER
I20260620 03:11:15.257341 22852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } } peers { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } } peers { permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } }
I20260620 03:11:15.258330 23020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } } peers { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } } peers { permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } } }
I20260620 03:11:15.258736 23020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:11:15.259558 22852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:11:15.259682 22827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 03:11:15.259888 22827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:11:15.262334 22827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } } peers { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } } peers { permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } }
I20260620 03:11:15.264591 22827 leader_election.cc:288] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [CANDIDATE]: Term 1 election: Requested vote from peers bd2d848849ea40de8642a880d98295a9 (127.0.97.61:33055), 14753a319fe446aead99bf1927d7b323 (127.0.97.60:41719)
I20260620 03:11:15.264554 22975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "14753a319fe446aead99bf1927d7b323"
I20260620 03:11:15.265079 22975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:11:15.269098 22975 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 in term 1.
I20260620 03:11:15.269879 22802 leader_election.cc:302] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14753a319fe446aead99bf1927d7b323, 7a4ea2f4b1d5426c8f05bfc2dcc35bd0; no voters:
I20260620 03:11:15.270491 23020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:11:15.279980 23020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a4ea2f4b1d5426c8f05bfc2dcc35bd0, State: Running, Role: LEADER
I20260620 03:11:15.280675 23020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } } peers { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } } peers { permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } }
I20260620 03:11:15.290310 23025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a4ea2f4b1d5426c8f05bfc2dcc35bd0. Latest consensus state: current_term: 1 leader_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } } peers { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } } peers { permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } } }
I20260620 03:11:15.290736 23025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:15.291610 22926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } attempt: 5
I20260620 03:11:15.292101 23033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 03:11:15 dist-test-slave-xv68 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260620 03:11:15.310879 22954 tablet.cc:2369] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:11:15.312264 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.292272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41719 (local address 127.0.0.1:39468)
0620 03:11:15.292750 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:15.292773 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 03:11:15.292900 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:11:15.293887 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:11:15.293896 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:11:15.294431 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:11:15.294662 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:15.294675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.296355 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.296364 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:15.298177 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:15.298195 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.300158 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.300167 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:15.300424 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.301152 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:11:15.301184 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:11:15.304698 (+ 3514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:11:15.309384 (+ 4686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:15.309390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:15.309396 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:15.309925 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:15.310359 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:15.310363 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:15.310366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:15.310526 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:15.311250 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:11:15.311255 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:11:15.311434 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 03:11:15.311596 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 03:11:15.311890 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304}
I20260620 03:11:15.313191 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.293145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39468 (local address 127.0.97.60:41719)
0620 03:11:15.293324 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.293330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.293349 (+ 19us) server_negotiation.cc:407] Connection header received
0620 03:11:15.293414 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.293419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.293524 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.293672 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.294840 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.296157 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.298376 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.299322 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.299461 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.304954 (+ 5493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:15.304963 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:15.304970 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:15.305020 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:15.309207 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.310068 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.310073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.310077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.310225 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.310676 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.310682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.310687 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.311049 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:15.311178 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:15.312640 (+ 1462us) server_negotiation.cc:299] Negotiation successful
0620 03:11:15.312860 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 03:11:15.314152 23033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 03:11:15.317873 22806 tablet.cc:2369] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:11:15.318849 22926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9: Bootstrap starting.
I20260620 03:11:15.323032 22926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9: Neither blocks nor log segments found. Creating new log.
I20260620 03:11:15.324795 22926 log.cc:824] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9: Log is configured to *not* fsync() on all Append() calls
I20260620 03:11:15.327605 23039 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 03:11:15.337999 22975 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323 [term 1 FOLLOWER]: Refusing update from remote peer 7a4ea2f4b1d5426c8f05bfc2dcc35bd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:11:15.339248 23025 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:11:15.340801 22926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9: No bootstrap required, opened a new log
I20260620 03:11:15.331786 22785 master_runner.cc:186] Error getting master registration for 127.0.97.61:33055: OK, instance_id { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" instance_seqno: 1781925074566883 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 03:11:15.346657 22785 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 03:11:15.361250 22926 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } } peers { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } } peers { permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } }
I20260620 03:11:15.361830 22926 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2d848849ea40de8642a880d98295a9, State: Initialized, Role: FOLLOWER
I20260620 03:11:15.362519 22926 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } } peers { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } } peers { permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } }
I20260620 03:11:15.368949 23006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a4ea2f4b1d5426c8f05bfc2dcc35bd0. Latest consensus state: current_term: 1 leader_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } } peers { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } } peers { permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } } }
I20260620 03:11:15.369349 23006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:11:15.371245 23040 mvcc.cc:205] Tried to move back new op lower bound from 7298765108565712896 to 7298765108355780608. Current Snapshot: MvccSnapshot[applied={T|T < 7298765108565712896}]
I20260620 03:11:15.374235 23045 mvcc.cc:205] Tried to move back new op lower bound from 7298765108565712896 to 7298765108355780608. Current Snapshot: MvccSnapshot[applied={T|T < 7298765108565712896}]
I20260620 03:11:15.382624 23025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } } peers { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } } peers { permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } } }
I20260620 03:11:15.382926 23025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:15.389505 23006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } } peers { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } } peers { permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } } }
I20260620 03:11:15.389787 23006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:11:15.394690 22926 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:11:15.395161 22900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:11:15.369880 23047 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } } peers { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } } peers { permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } } }
I20260620 03:11:15.396425 23047 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:11:15.397202 23033 catalog_manager.cc:1347] Generated new cluster ID: c3a523fe929f4439a4c1c97dd9cca4ab
I20260620 03:11:15.397282 23033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:11:15.397432 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.335067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39484 (local address 127.0.97.60:41719)
0620 03:11:15.335485 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.335494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.350891 (+ 15397us) server_negotiation.cc:407] Connection header received
0620 03:11:15.355569 (+ 4678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.355575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.355652 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.355747 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.360179 (+ 4432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.361183 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.379401 (+ 18218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.380301 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.380444 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.385057 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:15.385066 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:15.385072 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:15.385120 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:15.392674 (+ 7554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.393501 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.393506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.393509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.393670 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.394125 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.394132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.394135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.395516 (+ 1381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:15.395863 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:15.396634 (+ 771us) server_negotiation.cc:299] Negotiation successful
0620 03:11:15.396935 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266}
I20260620 03:11:15.399739 22901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bd2d848849ea40de8642a880d98295a9"
I20260620 03:11:15.399230 22900 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9 [term 1 FOLLOWER]: Refusing update from remote peer 7a4ea2f4b1d5426c8f05bfc2dcc35bd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:11:15.403712 22901 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 in term 1.
I20260620 03:11:15.404340 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.339448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48178 (local address 127.0.97.61:33055)
0620 03:11:15.339794 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.339799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.347751 (+ 7952us) server_negotiation.cc:407] Connection header received
0620 03:11:15.347916 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.347922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.348005 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.348108 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.353489 (+ 5381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.354687 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.361445 (+ 6758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.362151 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.362254 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.377862 (+ 15608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:15.377870 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:15.377875 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:15.377919 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:15.387439 (+ 9520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.391368 (+ 3929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.391374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.391378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.391526 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.392030 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.392037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.392042 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.393259 (+ 1217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:15.393517 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:15.394658 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0620 03:11:15.395030 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260620 03:11:15.404855 23020 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:11:15.405471 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.339653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.0.97.62:43615)
0620 03:11:15.339812 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.339817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.358349 (+ 18532us) server_negotiation.cc:407] Connection header received
0620 03:11:15.359158 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.359165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.359248 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.359344 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.369000 (+ 9656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.370056 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.379803 (+ 9747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.380583 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.380686 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.386586 (+ 5900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:15.386593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:15.386598 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:15.386639 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:15.390339 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.391343 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.391348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.391351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.391527 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:15.392322 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:15.392331 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:15.392336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:15.392721 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:15.392953 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:15.404816 (+ 11863us) server_negotiation.cc:299] Negotiation successful
0620 03:11:15.405114 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 03:11:15.461421 23047 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a4ea2f4b1d5426c8f05bfc2dcc35bd0. Latest consensus state: current_term: 1 leader_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } } peers { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } } peers { permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } } }
I20260620 03:11:15.461696 23047 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:11:15.462137 23054 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } } peers { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } } peers { permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } } }
I20260620 03:11:15.462428 23054 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:11:15.506830 23020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ea2f4b1d5426c8f05bfc2dcc35bd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43615 } } peers { permanent_uuid: "bd2d848849ea40de8642a880d98295a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33055 } } peers { permanent_uuid: "14753a319fe446aead99bf1927d7b323" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41719 } } }
I20260620 03:11:15.507244 23020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0 [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:15.514734 23033 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:11:15.517078 23033 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:11:15.535583 23033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a4ea2f4b1d5426c8f05bfc2dcc35bd0: Generated new TSK 0
I20260620 03:11:15.537220 23033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:11:15.541981 23064 catalog_manager.cc:1259] Loaded cluster ID: c3a523fe929f4439a4c1c97dd9cca4ab
I20260620 03:11:15.542074 23064 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9: loading cluster ID for follower catalog manager: success
I20260620 03:11:15.545509 23064 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9: acquiring CA information for follower catalog manager: success
I20260620 03:11:15.547974 23064 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bd2d848849ea40de8642a880d98295a9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 03:11:15.674919 22930 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:15.655341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.0.97.61:33055)
0620 03:11:15.655566 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.655575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.672293 (+ 16718us) server_negotiation.cc:407] Connection header received
0620 03:11:15.673304 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.673312 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.673418 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.673496 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.674691 (+ 1195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48190: BlockingRecv error: recv got EOF from 127.0.0.1:48190 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 03:11:15.678062 23003 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:15.657356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.0.97.60:41719)
0620 03:11:15.657558 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.657565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.676172 (+ 18607us) server_negotiation.cc:407] Connection header received
0620 03:11:15.676578 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.676584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.676679 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.676755 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.677830 (+ 1075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39490: BlockingRecv error: recv got EOF from 127.0.0.1:39490 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 03:11:15.680162 22856 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:15.670623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.0.97.62:43615)
0620 03:11:15.670841 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.670847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.678603 (+ 7756us) server_negotiation.cc:407] Connection header received
0620 03:11:15.678846 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.678852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.678956 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.679029 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.679929 (+ 900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59050: BlockingRecv error: recv got EOF from 127.0.0.1:59050 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 03:11:15.723721 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.708048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48194 (local address 127.0.97.61:33055)
0620 03:11:15.708253 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.708260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.715835 (+ 7575us) server_negotiation.cc:407] Connection header received
0620 03:11:15.716354 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.716360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.716575 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.716730 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:15.717406 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.718590 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.720718 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.721405 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.721500 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.722229 (+ 729us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:15.722866 (+ 637us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:15.722978 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:15.723217 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 03:11:15.723332 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 03:11:15.770033 23003 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:15.757145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39502 (local address 127.0.97.60:41719)
0620 03:11:15.757332 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.757339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.767953 (+ 10614us) server_negotiation.cc:407] Connection header received
0620 03:11:15.768458 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.768464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.768601 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.768696 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.769736 (+ 1040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39502: BlockingRecv error: recv got EOF from 127.0.0.1:39502 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:11:15.811628 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.804274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48210 (local address 127.0.97.61:33055)
0620 03:11:15.804515 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.804521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.805071 (+ 550us) server_negotiation.cc:407] Connection header received
0620 03:11:15.805315 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.805323 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.805414 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.805515 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:15.806039 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.807287 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.809315 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.810029 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.810128 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.810745 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:15.810910 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:15.810973 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:15.811211 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 03:11:15.811317 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 03:11:15.821424 23003 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:15.819132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.0.97.60:41719)
0620 03:11:15.819324 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.819330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.819831 (+ 501us) server_negotiation.cc:407] Connection header received
0620 03:11:15.820091 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.820097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.820189 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.820261 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.821206 (+ 945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39510: BlockingRecv error: recv got EOF from 127.0.0.1:39510 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 03:11:15.836390 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.828990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.0.97.61:33055)
0620 03:11:15.829155 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.829160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.829605 (+ 445us) server_negotiation.cc:407] Connection header received
0620 03:11:15.829869 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.829875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.829952 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.830013 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:15.830623 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.831802 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.834040 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.834798 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.834912 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.835456 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:15.835608 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:15.835716 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:15.835950 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 03:11:15.836051 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 03:11:15.844988 23003 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:15.842677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39514 (local address 127.0.97.60:41719)
0620 03:11:15.842844 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.842852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.843367 (+ 515us) server_negotiation.cc:407] Connection header received
0620 03:11:15.843643 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.843649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.843764 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.843838 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.844750 (+ 912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39514: BlockingRecv error: recv got EOF from 127.0.0.1:39514 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 03:11:15.867920 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.860854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48230 (local address 127.0.97.61:33055)
0620 03:11:15.861099 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.861105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.861616 (+ 511us) server_negotiation.cc:407] Connection header received
0620 03:11:15.861860 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.861867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.861965 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.862091 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:15.862536 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.863761 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.865693 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.866421 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.866550 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.867060 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:15.867214 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:15.867364 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:15.867512 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 03:11:15.867600 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260620 03:11:15.877409 23003 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:15.875446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39528 (local address 127.0.97.60:41719)
0620 03:11:15.875707 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.875715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.875825 (+ 110us) server_negotiation.cc:407] Connection header received
0620 03:11:15.876096 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.876102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.876212 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.876320 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.877143 (+ 823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39528: BlockingRecv error: recv got EOF from 127.0.0.1:39528 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
I20260620 03:11:15.895360 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.888480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48246 (local address 127.0.97.61:33055)
0620 03:11:15.888669 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.888674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.889165 (+ 491us) server_negotiation.cc:407] Connection header received
0620 03:11:15.889354 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.889361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.889463 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.889570 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:15.890109 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.891316 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.893242 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.893853 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.893978 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.894477 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:15.894634 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:15.894724 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:15.894890 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 03:11:15.894977 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 03:11:15.907094 23003 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:15.904920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39540 (local address 127.0.97.60:41719)
0620 03:11:15.905104 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.905110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.905607 (+ 497us) server_negotiation.cc:407] Connection header received
0620 03:11:15.905787 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.905795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.905902 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.906010 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.906878 (+ 868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39540: BlockingRecv error: recv got EOF from 127.0.0.1:39540 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:11:15.933841 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.927428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.0.97.61:33055)
0620 03:11:15.927635 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.927642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.928169 (+ 527us) server_negotiation.cc:407] Connection header received
0620 03:11:15.928368 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.928372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.928444 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.928523 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:15.929017 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.929847 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.931931 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.932444 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.932551 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.933074 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:15.933187 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:15.933268 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:15.933434 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 03:11:15.933510 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 03:11:15.942498 23003 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:15.940480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39546 (local address 127.0.97.60:41719)
0620 03:11:15.940649 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.940655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.941007 (+ 352us) server_negotiation.cc:407] Connection header received
0620 03:11:15.941172 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.941177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.941261 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.941349 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.942235 (+ 886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39546: BlockingRecv error: recv got EOF from 127.0.0.1:39546 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 03:11:15.984053 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:15.978054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.0.97.61:33055)
0620 03:11:15.978295 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.978301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.978673 (+ 372us) server_negotiation.cc:407] Connection header received
0620 03:11:15.978864 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.978871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.978958 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.979084 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:15.979493 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.980633 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.981977 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:15.982709 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:15.982862 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:15.983214 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:15.983371 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:15.983464 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:15.983575 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 03:11:15.983702 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 03:11:15.992846 23003 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:15.990712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39548 (local address 127.0.97.60:41719)
0620 03:11:15.990924 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:15.990931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:15.991425 (+ 494us) server_negotiation.cc:407] Connection header received
0620 03:11:15.991628 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:15.991635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:15.991735 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:15.991855 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:15.992585 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39548: BlockingRecv error: recv got EOF from 127.0.0.1:39548 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 03:11:16.074779 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:16.068533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48284 (local address 127.0.97.61:33055)
0620 03:11:16.068764 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:16.068770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:16.069121 (+ 351us) server_negotiation.cc:407] Connection header received
0620 03:11:16.069325 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:16.069329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:16.069399 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:16.069483 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:16.069958 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:16.070830 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:16.072989 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:16.073525 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:16.073620 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:16.074092 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:16.074200 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:16.074269 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:16.074411 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 03:11:16.074474 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260620 03:11:16.096843 23003 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:16.081450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39550 (local address 127.0.97.60:41719)
0620 03:11:16.081667 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:16.081675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:16.095282 (+ 13607us) server_negotiation.cc:407] Connection header received
0620 03:11:16.095509 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:16.095515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:16.095603 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:16.095701 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:16.096633 (+ 932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39550: BlockingRecv error: recv got EOF from 127.0.0.1:39550 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 03:11:16.181357 23017 catalog_manager.cc:1259] Loaded cluster ID: c3a523fe929f4439a4c1c97dd9cca4ab
I20260620 03:11:16.181495 23017 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323: loading cluster ID for follower catalog manager: success
I20260620 03:11:16.184010 23017 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323: acquiring CA information for follower catalog manager: success
I20260620 03:11:16.185771 23017 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 14753a319fe446aead99bf1927d7b323: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 03:11:16.233811 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:16.228853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48298 (local address 127.0.97.61:33055)
0620 03:11:16.229099 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:16.229106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:16.229288 (+ 182us) server_negotiation.cc:407] Connection header received
0620 03:11:16.229493 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:16.229498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:16.229567 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:16.229675 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:16.230030 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:16.230736 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:16.232175 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:16.232615 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:16.232731 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:16.233108 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:16.233233 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:16.233302 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:16.233426 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 03:11:16.233489 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 03:11:16.247192 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:16.241799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39554 (local address 127.0.97.60:41719)
0620 03:11:16.242043 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:16.242051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:16.242533 (+ 482us) server_negotiation.cc:407] Connection header received
0620 03:11:16.242660 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:16.242664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:16.242811 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:16.242904 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:16.243259 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:16.244029 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:16.245200 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:16.245600 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:16.245718 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:16.246156 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:16.246728 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:16.246834 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:16.246904 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 03:11:16.246964 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 03:11:16.264377 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:16.253466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.0.97.61:33055)
0620 03:11:16.253914 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:16.253923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:16.257610 (+ 3687us) server_negotiation.cc:407] Connection header received
0620 03:11:16.257819 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:16.257828 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:16.257944 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:16.258049 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:16.258839 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:16.260019 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:16.262207 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:16.262829 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:16.262923 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:16.263560 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:16.263697 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:16.263764 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:16.264004 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 03:11:16.264083 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289}
I20260620 03:11:16.265192 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:16.253691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.0.97.60:41719)
0620 03:11:16.254093 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:16.254102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:16.258670 (+ 4568us) server_negotiation.cc:407] Connection header received
0620 03:11:16.259080 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:16.259085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:16.259158 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:16.259214 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:16.259936 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:16.260994 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:16.263066 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:16.263713 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:16.263806 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:16.264396 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:16.264510 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:16.264575 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:16.264870 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 03:11:16.264943 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260620 03:11:16.272739 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:16.253126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.0.97.62:43615)
0620 03:11:16.267898 (+ 14772us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:16.267903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:16.267921 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:11:16.268033 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:16.268037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:16.268231 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:16.268399 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:16.268700 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:16.269369 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:16.270654 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:16.271162 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:16.271281 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:16.271795 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:16.272158 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:16.272266 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:16.272392 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 03:11:16.272476 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14632,"thread_start_us":14553,"threads_started":1}
I20260620 03:11:16.278048 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 22785
I20260620 03:11:16.278965 22847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:11:16.387380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 22785
I20260620 03:11:16.408294 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 22859
I20260620 03:11:16.409339 22921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:11:16.514181 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 22859
I20260620 03:11:16.534036 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 22933
I20260620 03:11:16.535115 22995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:11:16.642030 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 22933
2026-06-20T03:11:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4367 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:11:16 dist-test-slave-xv68 krb5kdc[23178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:11:16 dist-test-slave-xv68 krb5kdc[23178](info): set up 2 sockets
Jun 20 03:11:16 dist-test-slave-xv68 krb5kdc[23178](info): commencing operation
krb5kdc: starting...
W20260620 03:11:17.767948 388 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/test-user.keytab.
Jun 20 03:11:17 dist-test-slave-xv68 krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:11:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:11:18.012801 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46521
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46521,127.0.97.61:46697,127.0.97.60:37319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:11:18.337191 23191 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 03:11:18.337478 23191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:11:18.337548 23191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:11:18.337630 23191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:11:18.344663 23191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:11:18.344751 23191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:11:18.344807 23191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:11:18.344857 23191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:11:18.344918 23191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:11:18.354306 23191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46521,127.0.97.61:46697,127.0.97.60:37319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:11:18.355453 23191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:11:18.356606 23191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:11:18.368037 23196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:18.367971 23199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:18.368067 23197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:11:18.369349 23191 server_base.cc:1034] running on GCE node
I20260620 03:11:18.370026 23191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:11:18.371227 23191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:11:18.372473 23191 hybrid_clock.cc:648] HybridClock initialized: now 1781925078372411 us; error 28 us; skew 500 ppm
I20260620 03:11:18.372802 23191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:11:18 dist-test-slave-xv68 krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:11:18.380064 23191 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:11:18.381454 23191 webserver.cc:460] Webserver started at http://127.0.97.62:44045/ using document root <none> and password file <none>
I20260620 03:11:18.382097 23191 fs_manager.cc:359] Metadata directory not provided
I20260620 03:11:18.382187 23191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:11:18.382436 23191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:11:18.385380 23191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-0/data/instance:
uuid: "6fc7219035c04953b92719b2726402c0"
format_stamp: "Formatted at 2026-06-20 03:11:18 on dist-test-slave-xv68"
server_key: "de306866666196b37f958afcc2e0a077"
server_key_iv: "aa092641ad4993a89643de4bcaecb374"
server_key_version: "encryptionkey@0"
I20260620 03:11:18.386071 23191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-0/wal/instance:
uuid: "6fc7219035c04953b92719b2726402c0"
format_stamp: "Formatted at 2026-06-20 03:11:18 on dist-test-slave-xv68"
server_key: "de306866666196b37f958afcc2e0a077"
server_key_iv: "aa092641ad4993a89643de4bcaecb374"
server_key_version: "encryptionkey@0"
I20260620 03:11:18.390940 23191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 03:11:18.394338 23206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:11:18.395814 23191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:11:18.395973 23191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-0/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-0/wal
uuid: "6fc7219035c04953b92719b2726402c0"
format_stamp: "Formatted at 2026-06-20 03:11:18 on dist-test-slave-xv68"
server_key: "de306866666196b37f958afcc2e0a077"
server_key_iv: "aa092641ad4993a89643de4bcaecb374"
server_key_version: "encryptionkey@0"
I20260620 03:11:18.396188 23191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:11:18.411242 23191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:11:18.415205 23191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:11:18.415534 23191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:11:18.434993 23191 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46521
I20260620 03:11:18.435017 23257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46521 every 8 connection(s)
I20260620 03:11:18.436435 23191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:11:18.440029 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:11:18.445107 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 23191
I20260620 03:11:18.445302 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-0/wal/instance
I20260620 03:11:18.445751 388 external_mini_cluster.cc:1428] Setting key f41a424c4c4bbc9955bfa0d6e8ca8a5d
I20260620 03:11:18.446441 23258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:11:18 dist-test-slave-xv68 krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260620 03:11:18.472983 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:18.449695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46521 (local address 127.0.0.1:39820)
0620 03:11:18.450781 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:18.450819 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 03:11:18.452565 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:11:18.453361 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:11:18.453367 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:11:18.454040 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:11:18.454360 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:18.454376 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:18.455977 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:18.455983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:18.457029 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:18.457040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:18.457921 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:18.457928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:18.458101 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:18.458856 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:11:18.458890 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:11:18.461929 (+ 3039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:11:18.468887 (+ 6958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:18.468898 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:18.468910 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:18.469280 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:18.469670 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:18.469676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:18.469683 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:18.469851 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:18.470577 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:11:18.470585 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:11:18.470815 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 03:11:18.471055 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 03:11:18.471700 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":127,"thread_start_us":702,"threads_started":1}
I20260620 03:11:18.473003 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:18.449695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39820 (local address 127.0.97.62:46521)
0620 03:11:18.450503 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:18.450524 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:18.450939 (+ 415us) server_negotiation.cc:407] Connection header received
0620 03:11:18.452916 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:18.452921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:18.453136 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:18.453327 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:18.454563 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:18.455821 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:18.457166 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:18.457693 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:18.457899 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:18.462212 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:18.462225 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:18.462270 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:18.462312 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:18.468731 (+ 6419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:18.469409 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:18.469418 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:18.469421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:18.469565 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:18.469952 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:18.469959 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:18.469963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:18.470450 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:18.470657 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:18.471203 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 03:11:18.471560 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":364,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 03:11:18.476207 23258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:11:18.477264 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:18.476887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46697 (local address 127.0.0.1:53702)
0620 03:11:18.477049 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:18.477158 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 03:11:18.477533 23209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46697: connect: Connection refused (error 111)
I20260620 03:11:18.479043 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:18.478720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46697 (local address 127.0.0.1:53716)
0620 03:11:18.478854 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:18.478923 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 03:11:18.479256 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46697: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46697: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 03:11:18.501668 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:46697
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46521,127.0.97.61:46697,127.0.97.60:37319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:11:18.528604 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } attempt: 1
I20260620 03:11:18.529778 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:18.529218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46697 (local address 127.0.0.1:53732)
0620 03:11:18.529468 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:18.529588 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 03:11:18.530848 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:18.530422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46697 (local address 127.0.0.1:53734)
0620 03:11:18.530577 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:18.530671 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 03:11:18.531069 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46697: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46697: connect: Connection refused (error 111)
I20260620 03:11:18.599427 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } attempt: 2
I20260620 03:11:18.600724 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:18.600121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46697 (local address 127.0.0.1:53736)
0620 03:11:18.600406 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:18.600542 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 03:11:18.601851 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:18.601436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46697 (local address 127.0.0.1:53744)
0620 03:11:18.601586 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:18.601689 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 03:11:18.602057 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46697: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46697: connect: Connection refused (error 111)
I20260620 03:11:18.693339 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } attempt: 3
I20260620 03:11:18.694585 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:18.694052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46697 (local address 127.0.0.1:53758)
0620 03:11:18.694297 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:18.694406 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 03:11:18.695806 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:18.695398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46697 (local address 127.0.0.1:53764)
0620 03:11:18.695549 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:18.695646 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 03:11:18.695996 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46697: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46697: connect: Connection refused (error 111)
W20260620 03:11:18.824072 23265 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 03:11:18.824398 23265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:11:18.824512 23265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:11:18.824661 23265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:11:18.831784 23265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:11:18.831861 23265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:11:18.831949 23265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:11:18.832029 23265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:11:18.832121 23265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:11:18.839246 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } attempt: 4
I20260620 03:11:18.840508 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:18.839918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46697 (local address 127.0.0.1:53766)
0620 03:11:18.840238 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:18.840351 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260620 03:11:18.841501 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:18.841113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46697 (local address 127.0.0.1:53782)
0620 03:11:18.841254 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:18.841357 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 03:11:18.841749 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46697: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46697: connect: Connection refused (error 111)
I20260620 03:11:18.842700 23265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46521,127.0.97.61:46697,127.0.97.60:37319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:46697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:11:18.844060 23265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:11:18.845291 23265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:11:18.856443 23270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:18.856521 23271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:18.856541 23273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:11:18.857424 23265 server_base.cc:1034] running on GCE node
I20260620 03:11:18.858152 23265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:11:18.859411 23265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:11:18.860608 23265 hybrid_clock.cc:648] HybridClock initialized: now 1781925078860558 us; error 32 us; skew 500 ppm
I20260620 03:11:18.860919 23265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:11:18 dist-test-slave-xv68 krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:11:18.863924 23265 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 03:11:18.865291 23265 webserver.cc:460] Webserver started at http://127.0.97.61:43569/ using document root <none> and password file <none>
I20260620 03:11:18.865934 23265 fs_manager.cc:359] Metadata directory not provided
I20260620 03:11:18.866053 23265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:11:18.866348 23265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:11:18.869169 23265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-1/data/instance:
uuid: "020b1810c88f410a8b3c59206c8edfd5"
format_stamp: "Formatted at 2026-06-20 03:11:18 on dist-test-slave-xv68"
server_key: "7db2d6a41bddc6bcaa9ccd1ae6ce66f2"
server_key_iv: "a7a21d0c900586edb93fa52067a80bda"
server_key_version: "encryptionkey@0"
I20260620 03:11:18.869926 23265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-1/wal/instance:
uuid: "020b1810c88f410a8b3c59206c8edfd5"
format_stamp: "Formatted at 2026-06-20 03:11:18 on dist-test-slave-xv68"
server_key: "7db2d6a41bddc6bcaa9ccd1ae6ce66f2"
server_key_iv: "a7a21d0c900586edb93fa52067a80bda"
server_key_version: "encryptionkey@0"
I20260620 03:11:18.874650 23265 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 03:11:18.877867 23280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:11:18.879030 23265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 03:11:18.879199 23265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-1/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-1/wal
uuid: "020b1810c88f410a8b3c59206c8edfd5"
format_stamp: "Formatted at 2026-06-20 03:11:18 on dist-test-slave-xv68"
server_key: "7db2d6a41bddc6bcaa9ccd1ae6ce66f2"
server_key_iv: "a7a21d0c900586edb93fa52067a80bda"
server_key_version: "encryptionkey@0"
I20260620 03:11:18.879447 23265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:11:18.896945 23265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:11:18.901118 23265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:11:18.901368 23265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:11:18.918462 23265 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:46697
I20260620 03:11:18.918484 23331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:46697 every 8 connection(s)
I20260620 03:11:18.919747 23265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-1/data/info.pb
I20260620 03:11:18.923265 23332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:11:18.923659 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 23265
I20260620 03:11:18.923835 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-1/wal/instance
I20260620 03:11:18.924242 388 external_mini_cluster.cc:1428] Setting key 5798fc8e31f7ec9680b6e730cce44cd8
I20260620 03:11:18.929595 23332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:11:18 dist-test-slave-xv68 krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260620 03:11:18.950412 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:18.932106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39830 (local address 127.0.97.62:46521)
0620 03:11:18.932343 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:18.932354 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:18.933485 (+ 1131us) server_negotiation.cc:407] Connection header received
0620 03:11:18.935336 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:18.935344 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:18.935438 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:18.935552 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:18.936916 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:18.938082 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:18.939617 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:18.940362 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:18.940508 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:18.944428 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:18.944433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:18.944436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:18.944470 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:18.947126 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:18.947927 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:18.947931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:18.947934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:18.948052 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:18.948405 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:18.948408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:18.948410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:18.948724 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:18.948847 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:18.949730 (+ 883us) server_negotiation.cc:299] Negotiation successful
0620 03:11:18.949988 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 03:11:18.951067 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:18.932546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46521 (local address 127.0.0.1:39830)
0620 03:11:18.933317 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:18.933359 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 03:11:18.935047 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:11:18.935741 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:11:18.935748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:11:18.936235 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:11:18.936738 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:18.936754 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:18.938235 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:18.938245 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:18.939474 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:18.939486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:18.940569 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:18.940574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:18.940779 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:18.941437 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:11:18.941458 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:11:18.944191 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:11:18.947257 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:18.947263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:18.947275 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:18.947819 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:18.948143 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:18.948147 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:18.948149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:18.948309 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:18.948860 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:11:18.948865 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:11:18.949297 (+ 432us) client_negotiation.cc:769] Sending connection context
0620 03:11:18.949591 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 03:11:18.949967 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":349,"threads_started":1}
I20260620 03:11:18.952759 23332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:11:18 dist-test-slave-xv68 krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:11:18.967427 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:18.953510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46697 (local address 127.0.0.1:53798)
0620 03:11:18.953731 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:18.953764 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 03:11:18.953868 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:11:18.954749 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:11:18.954757 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:11:18.955252 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:11:18.955483 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:18.955496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:18.956922 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:18.956930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:18.957822 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:18.957833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:18.958724 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:18.958733 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:18.958976 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:18.959577 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:11:18.959604 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:11:18.961982 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:11:18.965082 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:18.965090 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:18.965094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:18.965453 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:18.965793 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:18.965800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:18.965806 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:18.965929 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:18.966476 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:11:18.966480 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:11:18.966625 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 03:11:18.966760 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 03:11:18.967020 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 03:11:18.967483 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:18.953712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53798 (local address 127.0.97.61:46697)
0620 03:11:18.954347 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:18.954353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:18.954377 (+ 24us) server_negotiation.cc:407] Connection header received
0620 03:11:18.954421 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:18.954426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:18.954580 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:18.954695 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:18.955658 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:18.956770 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:18.957956 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:18.958567 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:18.958674 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:18.962183 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:18.962193 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:18.962206 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:18.962232 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:18.964955 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:18.965577 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:18.965589 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:18.965591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:18.965693 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:18.966025 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:18.966032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:18.966036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:18.966351 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:18.966566 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:18.966911 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 03:11:18.967102 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":369,"threads_started":1}
I20260620 03:11:18.969812 23332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:11:18.970891 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:18.970482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40756)
0620 03:11:18.970630 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:18.970739 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 03:11:18.971112 23282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
I20260620 03:11:18.972616 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:18.972276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40758)
0620 03:11:18.972455 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:18.972522 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 03:11:18.972846 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37319: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 03:11:18.980419 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5VLrcV/build/asan/bin/kudu
/tmp/dist-test-task5VLrcV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37319
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46521,127.0.97.61:46697,127.0.97.60:37319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:11:19.022104 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } attempt: 1
I20260620 03:11:19.023327 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.022746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40774)
0620 03:11:19.022986 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.023098 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 03:11:19.024473 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.024032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40786)
0620 03:11:19.024216 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.024338 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 03:11:19.024756 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37319: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
I20260620 03:11:19.092993 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } attempt: 2
I20260620 03:11:19.094195 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.093662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40800)
0620 03:11:19.093889 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.094008 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 03:11:19.095208 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.094836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40810)
0620 03:11:19.094982 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.095066 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 03:11:19.095443 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37319: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
I20260620 03:11:19.141052 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } attempt: 5
Jun 20 03:11:19 dist-test-slave-xv68 krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:11:19.154187 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.141792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46697 (local address 127.0.0.1:53806)
0620 03:11:19.142012 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.142032 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:11:19.142163 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:11:19.142534 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:11:19.142538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:11:19.142969 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:11:19.143160 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:19.143169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.144207 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.144211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:19.145064 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:19.145075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.145800 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.145806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:19.146030 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:19.146638 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:11:19.146656 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:11:19.148760 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:11:19.151772 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:19.151777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:19.151782 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:19.152147 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:19.152506 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:19.152511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:19.152515 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:19.152655 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:19.153219 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:11:19.153223 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:11:19.153405 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 03:11:19.153543 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 03:11:19.153826 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 03:11:19.154186 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.141881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.0.97.61:46697)
0620 03:11:19.142058 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:19.142065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:19.142090 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:11:19.142298 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:19.142308 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:19.142396 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:19.142502 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:19.143350 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.144054 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.145212 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.145646 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.145774 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:19.148949 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:19.148954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:19.148957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:19.148989 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:19.151623 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.152268 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.152278 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.152280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.152398 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.152762 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.152768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.152771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.153107 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:19.153252 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:19.153641 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 03:11:19.153857 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 03:11:19.155517 23258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:11:19.156734 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.156185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40826)
0620 03:11:19.156408 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.156519 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 03:11:19.157783 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.157336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40840)
0620 03:11:19.157516 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.157617 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 03:11:19.157989 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37319: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
I20260620 03:11:19.186674 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } attempt: 3
I20260620 03:11:19.187924 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.187358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40844)
0620 03:11:19.187578 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.187702 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 03:11:19.189114 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.188616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40856)
0620 03:11:19.188884 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.188981 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260620 03:11:19.189311 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37319: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
I20260620 03:11:19.209220 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } attempt: 1
I20260620 03:11:19.210502 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.209961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40868)
0620 03:11:19.210180 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.210315 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 03:11:19.211655 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.211221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40884)
0620 03:11:19.211411 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.211504 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 03:11:19.211877 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37319: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
I20260620 03:11:19.280102 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } attempt: 2
I20260620 03:11:19.281427 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.280830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40892)
0620 03:11:19.281090 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.281215 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 03:11:19.282565 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.282087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40902)
0620 03:11:19.282357 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.282447 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
W20260620 03:11:19.282857 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37319: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
W20260620 03:11:19.303489 23339 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 03:11:19.303798 23339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:11:19.303884 23339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:11:19.304025 23339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:11:19.311120 23339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:11:19.311203 23339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:11:19.311300 23339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:11:19.311506 23339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:11:19.311604 23339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:11:19.321226 23339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46521,127.0.97.61:46697,127.0.97.60:37319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:11:19.322448 23339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:11:19.323776 23339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 03:11:19.332535 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } attempt: 4
I20260620 03:11:19.333901 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.333224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40908)
0620 03:11:19.333580 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.333685 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
W20260620 03:11:19.335033 23344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:11:19.335357 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.334878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40916)
0620 03:11:19.335105 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.335185 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260620 03:11:19.335281 23347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:11:19.335604 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37319: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
W20260620 03:11:19.335645 23345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:11:19.335898 23339 server_base.cc:1034] running on GCE node
I20260620 03:11:19.336566 23339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:11:19.337765 23339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:11:19.338973 23339 hybrid_clock.cc:648] HybridClock initialized: now 1781925079338919 us; error 35 us; skew 500 ppm
I20260620 03:11:19.339339 23339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:11:19 dist-test-slave-xv68 krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:11:19.342298 23339 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 03:11:19.343674 23339 webserver.cc:460] Webserver started at http://127.0.97.60:40697/ using document root <none> and password file <none>
I20260620 03:11:19.344342 23339 fs_manager.cc:359] Metadata directory not provided
I20260620 03:11:19.344463 23339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:11:19.344770 23339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:11:19.347687 23339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-2/data/instance:
uuid: "553b91f53545434681b840548b78d882"
format_stamp: "Formatted at 2026-06-20 03:11:19 on dist-test-slave-xv68"
server_key: "eac10ee9c59ad6a9bd99c1ed054b0588"
server_key_iv: "7fa8c4dba20256b89cff68b0fde546fe"
server_key_version: "encryptionkey@0"
I20260620 03:11:19.348446 23339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-2/wal/instance:
uuid: "553b91f53545434681b840548b78d882"
format_stamp: "Formatted at 2026-06-20 03:11:19 on dist-test-slave-xv68"
server_key: "eac10ee9c59ad6a9bd99c1ed054b0588"
server_key_iv: "7fa8c4dba20256b89cff68b0fde546fe"
server_key_version: "encryptionkey@0"
I20260620 03:11:19.353317 23339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 03:11:19.356544 23354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:11:19.357825 23339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:11:19.357992 23339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-2/data,/tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-2/wal
uuid: "553b91f53545434681b840548b78d882"
format_stamp: "Formatted at 2026-06-20 03:11:19 on dist-test-slave-xv68"
server_key: "eac10ee9c59ad6a9bd99c1ed054b0588"
server_key_iv: "7fa8c4dba20256b89cff68b0fde546fe"
server_key_version: "encryptionkey@0"
I20260620 03:11:19.358189 23339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:11:19.371747 23339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:11:19.375572 23339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:11:19.375821 23339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:11:19.389091 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } attempt: 3
I20260620 03:11:19.390307 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.389683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40924)
0620 03:11:19.389937 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.390052 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 03:11:19.391527 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.390985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40930)
0620 03:11:19.391231 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.391359 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260620 03:11:19.391757 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37319: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37319: connect: Connection refused (error 111)
I20260620 03:11:19.398012 23339 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:37319
I20260620 03:11:19.398038 23405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:37319 every 8 connection(s)
I20260620 03:11:19.400036 23339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-2/data/info.pb
I20260620 03:11:19.403358 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5VLrcV/build/asan/bin/kudu as pid 23339
I20260620 03:11:19.403538 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5VLrcV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924720365844-388-0/minicluster-data/master-2/wal/instance
I20260620 03:11:19.404078 23406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:11:19.404071 388 external_mini_cluster.cc:1428] Setting key c0eb24c3efb0fc8397b3ebc72f612fa2
I20260620 03:11:19.413331 23406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:11:19 dist-test-slave-xv68 krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 03:11:19 dist-test-slave-xv68 krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:11:19.427932 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.405790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40940 (local address 127.0.97.60:37319)
0620 03:11:19.406616 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:19.406628 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:19.406683 (+ 55us) server_negotiation.cc:407] Connection header received
0620 03:11:19.408607 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:19.408629 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:19.409013 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:19.409320 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:19.411025 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.412300 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.414080 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.414884 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.415025 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:19.419504 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:19.419511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:19.419521 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:19.419558 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:19.423219 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.423950 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.423957 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.423970 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.424160 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.424518 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.424523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.424527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.425114 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:19.425342 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:19.425915 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 03:11:19.426389 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":379,"threads_started":1}
I20260620 03:11:19.432425 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.416164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46521 (local address 127.0.0.1:39838)
0620 03:11:19.416940 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.416981 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 03:11:19.417139 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:11:19.417538 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:11:19.417543 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:11:19.418175 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:11:19.418461 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:19.418477 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.419787 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.419792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:19.421614 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:19.421631 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.422406 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.422416 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:19.422592 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:19.423178 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:11:19.423194 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:11:19.425462 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:11:19.429513 (+ 4051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:19.429524 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:19.429531 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:19.430088 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:19.430537 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:19.430542 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:19.430545 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:19.430680 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:19.431330 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:11:19.431338 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:11:19.431530 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 03:11:19.431726 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 03:11:19.432010 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":385,"threads_started":1}
I20260620 03:11:19.432466 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.416125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.0.97.62:46521)
0620 03:11:19.416345 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:19.416353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:19.417088 (+ 735us) server_negotiation.cc:407] Connection header received
0620 03:11:19.417274 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:19.417280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:19.417369 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:19.417479 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:19.418624 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.419635 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.421767 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.422193 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.422339 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:19.425728 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:19.425736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:19.425741 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:19.425784 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:19.429369 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.430219 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.430223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.430225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.430410 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.430792 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.430797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.430800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.431119 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:19.431354 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:19.431871 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 03:11:19.432081 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 03:11:19.433884 23406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:11:19 dist-test-slave-xv68 krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 03:11:19 dist-test-slave-xv68 krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:11:19.447638 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.431358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39852 (local address 127.0.97.62:46521)
0620 03:11:19.431909 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:19.431913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:19.431931 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:11:19.431968 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:19.431971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:19.432020 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:19.432139 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:19.433810 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.435076 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.436582 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.437245 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.437357 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:19.442067 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:19.442076 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:19.442082 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:19.442120 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:19.445099 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.445697 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.445703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.445706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.445828 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.446191 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.446196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.446200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.446462 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:19.446715 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:19.447006 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 03:11:19.447219 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":270,"threads_started":1}
I20260620 03:11:19.447801 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.434619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46697 (local address 127.0.0.1:53822)
0620 03:11:19.434867 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.434889 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:11:19.434998 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:11:19.435355 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:11:19.435360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:11:19.435799 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:11:19.435963 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:19.435973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.437108 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.437113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:19.438394 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:19.438410 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.439393 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.439401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:19.439624 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:19.440196 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:11:19.440222 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:11:19.442169 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:11:19.445542 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:19.445548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:19.445552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:19.445896 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:19.446274 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:19.446279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:19.446282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:19.446415 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:19.446889 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:11:19.446893 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:11:19.447039 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 03:11:19.447168 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 03:11:19.447449 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260620 03:11:19.447842 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.434608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.0.97.61:46697)
0620 03:11:19.434803 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:19.434808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:19.434955 (+ 147us) server_negotiation.cc:407] Connection header received
0620 03:11:19.435098 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:19.435103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:19.435194 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:19.435351 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:19.436117 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.436968 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.438553 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.439180 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.439367 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:19.442357 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:19.442363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:19.442367 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:19.442404 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:19.445415 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.446025 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.446030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.446033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.446161 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.446514 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.446521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.446524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.446770 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:19.446921 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:19.447262 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 03:11:19.447482 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 03:11:19.448808 23406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:11:19 dist-test-slave-xv68 krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 03:11:19 dist-test-slave-xv68 krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:11:19.462386 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.449558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40954 (local address 127.0.97.60:37319)
0620 03:11:19.449731 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:19.449736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:19.449935 (+ 199us) server_negotiation.cc:407] Connection header received
0620 03:11:19.450077 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:19.450081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:19.450146 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:19.450248 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:19.450860 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.451843 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.453085 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.453726 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.453824 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:19.457134 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:19.457138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:19.457141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:19.457167 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:19.459727 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.460500 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.460503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.460506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.460606 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.461002 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.461005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.461007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.461248 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:19.461350 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:19.461822 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 03:11:19.462127 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":60}
I20260620 03:11:19.462416 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.449440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40954)
0620 03:11:19.449854 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.449873 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:11:19.449968 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:11:19.450268 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:11:19.450271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:11:19.450581 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:11:19.450705 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:19.450712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.451966 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.451971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:19.452955 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:19.452966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.453871 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.453876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:19.454092 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:19.454662 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:11:19.454677 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:11:19.456969 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:11:19.459852 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:19.459857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:19.459861 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:19.460395 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:19.460723 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:19.460729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:19.460733 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:19.460886 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:19.461365 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:11:19.461370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:11:19.461521 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 03:11:19.461674 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 03:11:19.461984 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321}
I20260620 03:11:19.464386 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.449651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.0.97.61:46697)
0620 03:11:19.449827 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:19.449832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:19.449849 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:11:19.449898 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:19.449904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:19.449973 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:19.450056 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:19.451444 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.452291 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.453678 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.454260 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.454374 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:19.458772 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:19.458777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:19.458780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:19.458813 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:19.461751 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.462541 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.462549 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.462553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.462677 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.463066 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.463072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.463075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.463431 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:19.463709 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:19.463941 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 03:11:19.464120 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 03:11:19.468079 23406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882: Bootstrap starting.
I20260620 03:11:19.468673 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:11:19.471917 23406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882: Neither blocks nor log segments found. Creating new log.
I20260620 03:11:19.473050 23406 log.cc:824] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882: Log is configured to *not* fsync() on all Append() calls
I20260620 03:11:19.471763 23233 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6fc7219035c04953b92719b2726402c0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39852
I20260620 03:11:19.476073 23406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882: No bootstrap required, opened a new log
I20260620 03:11:19.483070 23406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc7219035c04953b92719b2726402c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } peers { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } } peers { permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } }
I20260620 03:11:19.483510 23406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553b91f53545434681b840548b78d882, State: Initialized, Role: FOLLOWER
I20260620 03:11:19.483983 23406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc7219035c04953b92719b2726402c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } peers { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } } peers { permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } }
I20260620 03:11:19.484846 23413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc7219035c04953b92719b2726402c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } peers { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } } peers { permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } } }
I20260620 03:11:19.485167 23413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:11:19.486055 23406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 03:11:19.494690 23424 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 03:11:19.494839 23424 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 03:11:19.493296 23339 master_runner.cc:186] Error getting master registration for 127.0.97.62:46521: OK, instance_id { permanent_uuid: "6fc7219035c04953b92719b2726402c0" instance_seqno: 1781925078429501 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 03:11:19.504520 23339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 03:11:19.568965 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } attempt: 4
Jun 20 03:11:19 dist-test-slave-xv68 krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 03:11:19.585696 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.569676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40962)
0620 03:11:19.569920 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.569943 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 03:11:19.570079 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:11:19.570481 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:11:19.570486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:11:19.570984 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:11:19.571203 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:19.571217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.572740 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.572746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:19.574233 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:19.574249 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.575229 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.575235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:19.575491 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:19.576076 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:11:19.576101 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:11:19.578799 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:11:19.582817 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:19.582823 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:19.582828 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:19.583395 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:19.583766 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:19.583771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:19.583775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:19.583963 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:19.584646 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:11:19.584651 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:11:19.584828 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 03:11:19.584970 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 03:11:19.585300 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 03:11:19.585738 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.569740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40962 (local address 127.0.97.60:37319)
0620 03:11:19.570017 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:19.570034 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:19.570061 (+ 27us) server_negotiation.cc:407] Connection header received
0620 03:11:19.570224 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:19.570230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:19.570318 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:19.570424 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:19.571400 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.572581 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.574384 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.575080 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.575191 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:19.578991 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:19.578999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:19.579007 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:19.579044 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:19.582641 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.583518 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.583523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.583528 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.583653 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.584103 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.584112 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.584120 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.584522 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:19.584699 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:19.585098 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 03:11:19.585364 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 03:11:19.592006 23258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0: Bootstrap starting.
I20260620 03:11:19.595163 23258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0: Neither blocks nor log segments found. Creating new log.
I20260620 03:11:19.596241 23258 log.cc:824] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0: Log is configured to *not* fsync() on all Append() calls
I20260620 03:11:19.598932 23258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0: No bootstrap required, opened a new log
I20260620 03:11:19.604277 23258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc7219035c04953b92719b2726402c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } peers { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } } peers { permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } }
I20260620 03:11:19.604573 23258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc7219035c04953b92719b2726402c0, State: Initialized, Role: FOLLOWER
I20260620 03:11:19.604985 23258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc7219035c04953b92719b2726402c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } peers { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } } peers { permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } }
I20260620 03:11:19.605856 23427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc7219035c04953b92719b2726402c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } peers { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } } peers { permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } } }
I20260620 03:11:19.606230 23427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:11:19.606938 23258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:11:19.607103 23233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 03:11:19.607374 23233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:11:19.610934 23233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc7219035c04953b92719b2726402c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } peers { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } } peers { permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } }
I20260620 03:11:19.613607 23233 leader_election.cc:288] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [CANDIDATE]: Term 1 election: Requested vote from peers 020b1810c88f410a8b3c59206c8edfd5 (127.0.97.61:46697), 553b91f53545434681b840548b78d882 (127.0.97.60:37319)
I20260620 03:11:19.613631 23381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fc7219035c04953b92719b2726402c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "553b91f53545434681b840548b78d882"
I20260620 03:11:19.614123 23381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:11:19.617534 23381 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fc7219035c04953b92719b2726402c0 in term 1.
I20260620 03:11:19.618302 23208 leader_election.cc:302] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 553b91f53545434681b840548b78d882, 6fc7219035c04953b92719b2726402c0; no voters:
I20260620 03:11:19.620525 23427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [term 1 FOLLOWER]: Leader election won for term 1
W20260620 03:11:19.621102 23438 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 03:11:19.621233 23438 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 03:11:19.627667 23360 tablet.cc:2369] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:11:19.628335 23427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [term 1 LEADER]: Becoming Leader. State: Replica: 6fc7219035c04953b92719b2726402c0, State: Running, Role: LEADER
I20260620 03:11:19.629030 23427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc7219035c04953b92719b2726402c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } peers { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } } peers { permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } }
I20260620 03:11:19.634810 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } attempt: 5
Jun 20 03:11:19 dist-test-slave-xv68 krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781925078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 03:11:19.656729 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.635449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37319 (local address 127.0.0.1:40974)
0620 03:11:19.635837 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 03:11:19.635858 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:11:19.635975 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:11:19.636712 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:11:19.636720 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:11:19.637373 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:11:19.637639 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:19.637656 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.639515 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.639520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:19.641552 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:11:19.641571 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.643036 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.643045 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:11:19.643388 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:19.644253 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:11:19.644278 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:11:19.647991 (+ 3713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:11:19.653069 (+ 5078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:19.653077 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:19.653084 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:19.653846 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:19.654388 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:11:19.654396 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:11:19.654401 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:11:19.654599 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:11:19.655438 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:11:19.655444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:11:19.655641 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 03:11:19.655815 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 03:11:19.656174 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260}
I20260620 03:11:19.657994 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.635842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40974 (local address 127.0.97.60:37319)
0620 03:11:19.636066 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:19.636075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:19.636100 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:11:19.636181 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:19.636189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:19.636324 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:19.636470 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:19.637862 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.639329 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.641754 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.642648 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.642793 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:19.648263 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:19.648273 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:19.648279 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:19.648330 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:19.652846 (+ 4516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.654019 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.654025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.654032 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.654234 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.654740 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.654748 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.654754 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.655160 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:19.655367 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:19.657201 (+ 1834us) server_negotiation.cc:299] Negotiation successful
0620 03:11:19.657493 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 03:11:19.665202 23332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5: Bootstrap starting.
W20260620 03:11:19.668267 23212 tablet.cc:2369] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:11:19.671475 23441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fc7219035c04953b92719b2726402c0. Latest consensus state: current_term: 1 leader_uuid: "6fc7219035c04953b92719b2726402c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc7219035c04953b92719b2726402c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } peers { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } } peers { permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } } }
I20260620 03:11:19.671909 23441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:19.679955 23448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:11:19.682188 23332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5: Neither blocks nor log segments found. Creating new log.
I20260620 03:11:19.684228 23332 log.cc:824] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5: Log is configured to *not* fsync() on all Append() calls
I20260620 03:11:19.686614 23448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:11:19.674497 23191 master_runner.cc:186] Error getting master registration for 127.0.97.61:46697: OK, instance_id { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" instance_seqno: 1781925078913818 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 03:11:19.692116 23191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 03:11:19.698208 23332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5: No bootstrap required, opened a new log
I20260620 03:11:19.719257 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.667367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53834 (local address 127.0.97.61:46697)
0620 03:11:19.667565 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:19.667572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:19.697295 (+ 29723us) server_negotiation.cc:407] Connection header received
0620 03:11:19.697588 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:19.697594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:19.697672 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:19.697769 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:19.701510 (+ 3741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.702955 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.705470 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.706242 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.706379 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:19.709991 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:19.710000 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:19.710005 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:19.710054 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:19.714107 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.715509 (+ 1402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.715515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.715518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.715675 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.716167 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.716172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.716175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.716546 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:19.716779 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:19.718191 (+ 1412us) server_negotiation.cc:299] Negotiation successful
0620 03:11:19.718556 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 03:11:19.719343 23381 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882 [term 1 FOLLOWER]: Refusing update from remote peer 6fc7219035c04953b92719b2726402c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:11:19.721249 23441 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 03:11:19.721846 23332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc7219035c04953b92719b2726402c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } peers { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } } peers { permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } }
I20260620 03:11:19.722259 23332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 020b1810c88f410a8b3c59206c8edfd5, State: Initialized, Role: FOLLOWER
I20260620 03:11:19.722791 23332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc7219035c04953b92719b2726402c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } peers { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } } peers { permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } }
I20260620 03:11:19.738869 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.666784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39862 (local address 127.0.97.62:46521)
0620 03:11:19.667033 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:19.667043 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:19.694305 (+ 27262us) server_negotiation.cc:407] Connection header received
0620 03:11:19.696578 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:19.696586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:19.696678 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:19.696782 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:19.708694 (+ 11912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.709829 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.711761 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.712476 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.712576 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:19.717899 (+ 5323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:19.717907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:19.717913 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:19.717952 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:19.724639 (+ 6687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.725776 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.725785 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.725791 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.725971 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.726526 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.726534 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.726540 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.726982 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:19.727275 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:19.737960 (+ 10685us) server_negotiation.cc:299] Negotiation successful
0620 03:11:19.738324 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 03:11:19.756238 23413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fc7219035c04953b92719b2726402c0. Latest consensus state: current_term: 1 leader_uuid: "6fc7219035c04953b92719b2726402c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc7219035c04953b92719b2726402c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } peers { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } } peers { permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } } }
I20260620 03:11:19.756562 23413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:11:19.758787 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:19.698180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40980 (local address 127.0.97.60:37319)
0620 03:11:19.698372 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:19.698377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:19.713781 (+ 15404us) server_negotiation.cc:407] Connection header received
0620 03:11:19.714071 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:19.714078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:19.714166 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:19.714242 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:19.719873 (+ 5631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.720910 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.740940 (+ 20030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:19.741980 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:19.742121 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:19.749719 (+ 7598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:11:19.749729 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:11:19.749737 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:11:19.749792 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:11:19.754718 (+ 4926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.755711 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.755720 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.755725 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.755890 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:11:19.756345 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:11:19.756349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:11:19.756363 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:11:19.756925 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:11:19.757207 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:19.757845 (+ 638us) server_negotiation.cc:299] Negotiation successful
0620 03:11:19.758179 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 03:11:19.761428 23450 mvcc.cc:205] Tried to move back new op lower bound from 7298765126512115712 to 7298765126166945792. Current Snapshot: MvccSnapshot[applied={T|T < 7298765126512115712}]
I20260620 03:11:19.769044 23453 mvcc.cc:205] Tried to move back new op lower bound from 7298765126512115712 to 7298765126166945792. Current Snapshot: MvccSnapshot[applied={T|T < 7298765126512115712}]
I20260620 03:11:19.774389 23452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc7219035c04953b92719b2726402c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } peers { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } } peers { permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } } }
I20260620 03:11:19.776161 23452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:11:19.775990 23332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:11:19.776754 23306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:11:19.780433 23306 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5 [term 1 FOLLOWER]: Refusing update from remote peer 6fc7219035c04953b92719b2726402c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:11:19.781397 23427 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:11:19.776065 23307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fc7219035c04953b92719b2726402c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "020b1810c88f410a8b3c59206c8edfd5"
I20260620 03:11:19.784578 23413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fc7219035c04953b92719b2726402c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc7219035c04953b92719b2726402c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } peers { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } } peers { permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } } }
I20260620 03:11:19.784821 23413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:11:19.787741 23448 catalog_manager.cc:1347] Generated new cluster ID: fd221ae0939c4d25a16ba723d9b30ad4
I20260620 03:11:19.787844 23448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:11:19.803215 23427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fc7219035c04953b92719b2726402c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc7219035c04953b92719b2726402c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } peers { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } } peers { permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } } }
I20260620 03:11:19.803574 23427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:19.826413 23458 mvcc.cc:205] Tried to move back new op lower bound from 7298765126512115712 to 7298765126166945792. Current Snapshot: MvccSnapshot[applied={T|T < 7298765126512115712}]
I20260620 03:11:19.831398 23452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fc7219035c04953b92719b2726402c0. Latest consensus state: current_term: 1 leader_uuid: "6fc7219035c04953b92719b2726402c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc7219035c04953b92719b2726402c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } peers { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } } peers { permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } } }
I20260620 03:11:19.831696 23452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:11:19.832039 23460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fc7219035c04953b92719b2726402c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc7219035c04953b92719b2726402c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } peers { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } } peers { permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } } }
I20260620 03:11:19.832383 23460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:11:19.835634 23307 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6fc7219035c04953b92719b2726402c0 for term 1 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260620 03:11:19.871042 23441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fc7219035c04953b92719b2726402c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc7219035c04953b92719b2726402c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } peers { permanent_uuid: "020b1810c88f410a8b3c59206c8edfd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46697 } } peers { permanent_uuid: "553b91f53545434681b840548b78d882" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37319 } } }
I20260620 03:11:19.871419 23441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0 [sys.catalog]: This master's current role is: LEADER
I20260620 03:11:19.885039 23448 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:11:19.888607 23448 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:11:19.933710 23448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fc7219035c04953b92719b2726402c0: Generated new TSK 0
I20260620 03:11:19.934906 23448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:11:19.964355 23472 catalog_manager.cc:1259] Loaded cluster ID: fd221ae0939c4d25a16ba723d9b30ad4
I20260620 03:11:19.964473 23472 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5: loading cluster ID for follower catalog manager: success
I20260620 03:11:19.968334 23472 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5: acquiring CA information for follower catalog manager: success
I20260620 03:11:19.970820 23472 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 020b1810c88f410a8b3c59206c8edfd5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 03:11:20.073863 23411 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:20.050191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39878 (local address 127.0.97.62:46521)
0620 03:11:20.050411 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.050417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.071661 (+ 21244us) server_negotiation.cc:407] Connection header received
0620 03:11:20.072148 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.072155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.072281 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.072375 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:20.073552 (+ 1177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39878: BlockingRecv error: recv got EOF from 127.0.0.1:39878 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 03:11:20.076579 23336 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:20.056903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.0.97.61:46697)
0620 03:11:20.057082 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.057088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.074582 (+ 17494us) server_negotiation.cc:407] Connection header received
0620 03:11:20.075007 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.075014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.075130 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.075217 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:20.076343 (+ 1126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53850: BlockingRecv error: recv got EOF from 127.0.0.1:53850 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 03:11:20.079818 23409 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:20.077603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40986 (local address 127.0.97.60:37319)
0620 03:11:20.077773 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.077779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.078304 (+ 525us) server_negotiation.cc:407] Connection header received
0620 03:11:20.078527 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.078533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.078644 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.078721 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:20.079589 (+ 868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40986: BlockingRecv error: recv got EOF from 127.0.0.1:40986 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 03:11:20.101377 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:20.093460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53852 (local address 127.0.97.61:46697)
0620 03:11:20.093638 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.093644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.094133 (+ 489us) server_negotiation.cc:407] Connection header received
0620 03:11:20.094364 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.094370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.094568 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.094747 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:20.095181 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.096372 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.098399 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.099136 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.099268 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:20.099856 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:20.100531 (+ 675us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:20.100697 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:20.100936 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 03:11:20.101028 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 03:11:20.110512 23409 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:20.108524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.0.97.60:37319)
0620 03:11:20.108696 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.108701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.108722 (+ 21us) server_negotiation.cc:407] Connection header received
0620 03:11:20.108951 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.108957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.109101 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.109176 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:20.110272 (+ 1096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40996: BlockingRecv error: recv got EOF from 127.0.0.1:40996 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 03:11:20.123844 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:20.117085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.0.97.61:46697)
0620 03:11:20.117275 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.117280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.117762 (+ 482us) server_negotiation.cc:407] Connection header received
0620 03:11:20.117945 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.117950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.118085 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.118189 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:20.118638 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.119816 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.121756 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.122414 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.122545 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:20.123020 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:20.123167 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:20.123271 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:20.123445 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 03:11:20.123527 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 03:11:20.132921 23409 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:20.131379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41000 (local address 127.0.97.60:37319)
0620 03:11:20.131580 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.131586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.131605 (+ 19us) server_negotiation.cc:407] Connection header received
0620 03:11:20.131669 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.131673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.131758 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.131873 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:20.132700 (+ 827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41000: BlockingRecv error: recv got EOF from 127.0.0.1:41000 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:11:20.144767 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:20.137820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53864 (local address 127.0.97.61:46697)
0620 03:11:20.137983 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.137989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.138488 (+ 499us) server_negotiation.cc:407] Connection header received
0620 03:11:20.138659 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.138666 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.138745 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.138840 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:20.139311 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.140446 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.142395 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.143088 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.143219 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:20.143849 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:20.144022 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:20.144090 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:20.144340 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 03:11:20.144420 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 03:11:20.156749 23409 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:20.151831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41006 (local address 127.0.97.60:37319)
0620 03:11:20.152009 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.152015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.155221 (+ 3206us) server_negotiation.cc:407] Connection header received
0620 03:11:20.155468 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.155475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.155580 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.155698 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:20.156502 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41006: BlockingRecv error: recv got EOF from 127.0.0.1:41006 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 03:11:20.170296 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:20.163572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.0.97.61:46697)
0620 03:11:20.163764 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.163770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.164233 (+ 463us) server_negotiation.cc:407] Connection header received
0620 03:11:20.164431 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.164437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.164520 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.164620 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:20.165057 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.166236 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.168261 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.168968 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.169092 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:20.169585 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:20.169712 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:20.169807 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:20.169978 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 03:11:20.170055 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 03:11:20.179461 23409 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:20.177681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41018 (local address 127.0.97.60:37319)
0620 03:11:20.177945 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.177951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.177973 (+ 22us) server_negotiation.cc:407] Connection header received
0620 03:11:20.178047 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.178054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.178173 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.178293 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:20.179199 (+ 906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41018: BlockingRecv error: recv got EOF from 127.0.0.1:41018 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
I20260620 03:11:20.204342 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:20.197268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.0.97.61:46697)
0620 03:11:20.197468 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.197474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.197986 (+ 512us) server_negotiation.cc:407] Connection header received
0620 03:11:20.198194 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.198200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.198304 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.198419 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:20.198881 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.200109 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.202251 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.202906 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.203033 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:20.203523 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:20.203679 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:20.203777 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:20.203921 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 03:11:20.204008 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 03:11:20.217247 23409 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:20.211908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.0.97.60:37319)
0620 03:11:20.212097 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.212103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.215795 (+ 3692us) server_negotiation.cc:407] Connection header received
0620 03:11:20.215970 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.215977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.216078 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.216202 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:20.217005 (+ 803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41032: BlockingRecv error: recv got EOF from 127.0.0.1:41032 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 03:11:20.243328 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:20.236536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.0.97.61:46697)
0620 03:11:20.236722 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.236729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.237227 (+ 498us) server_negotiation.cc:407] Connection header received
0620 03:11:20.237420 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.237426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.237512 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.237613 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:20.238067 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.239250 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.241211 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.241897 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.242039 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:20.242537 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:20.242689 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:20.242776 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:20.242935 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 03:11:20.243024 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 03:11:20.252650 23409 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:20.250521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41036 (local address 127.0.97.60:37319)
0620 03:11:20.250695 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.250702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.251211 (+ 509us) server_negotiation.cc:407] Connection header received
0620 03:11:20.251395 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.251401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.251494 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.251597 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:20.252414 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41036: BlockingRecv error: recv got EOF from 127.0.0.1:41036 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 03:11:20.295174 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:20.288930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53910 (local address 127.0.97.61:46697)
0620 03:11:20.289131 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.289139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.289665 (+ 526us) server_negotiation.cc:407] Connection header received
0620 03:11:20.289868 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.289872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.289961 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.290044 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:20.290531 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.291362 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.293291 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.293796 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.293901 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:20.294416 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:20.294516 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:20.294589 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:20.294773 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 03:11:20.294836 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 03:11:20.303006 23409 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:20.300962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.0.97.60:37319)
0620 03:11:20.301179 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.301185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.301491 (+ 306us) server_negotiation.cc:407] Connection header received
0620 03:11:20.301665 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.301701 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.301774 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.301857 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:20.302762 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41042: BlockingRecv error: recv got EOF from 127.0.0.1:41042 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 03:11:20.377234 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:20.371046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53918 (local address 127.0.97.61:46697)
0620 03:11:20.371253 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.371260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.371658 (+ 398us) server_negotiation.cc:407] Connection header received
0620 03:11:20.371812 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.371818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.371914 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.372026 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:20.372465 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.373674 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.375103 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.375815 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.375967 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:20.376404 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:20.376558 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:20.376649 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:20.376838 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 03:11:20.376929 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 03:11:20.386281 23409 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:11:20.384319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.0.97.60:37319)
0620 03:11:20.384587 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.384591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.384712 (+ 121us) server_negotiation.cc:407] Connection header received
0620 03:11:20.384897 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.384901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.384971 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.385050 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:11:20.386000 (+ 950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41056: BlockingRecv error: recv got EOF from 127.0.0.1:41056 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
I20260620 03:11:20.496897 23424 catalog_manager.cc:1259] Loaded cluster ID: fd221ae0939c4d25a16ba723d9b30ad4
I20260620 03:11:20.496997 23424 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882: loading cluster ID for follower catalog manager: success
I20260620 03:11:20.499471 23424 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882: acquiring CA information for follower catalog manager: success
I20260620 03:11:20.501329 23424 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 553b91f53545434681b840548b78d882: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 03:11:20.522424 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:20.517394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53934 (local address 127.0.97.61:46697)
0620 03:11:20.517637 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.517644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.517912 (+ 268us) server_negotiation.cc:407] Connection header received
0620 03:11:20.518127 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.518132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.518211 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.518314 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:20.518701 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.519429 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.520767 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.521227 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.521360 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:20.521727 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:20.521840 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:20.521933 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:20.522073 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 03:11:20.522150 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 03:11:20.532706 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:20.527364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41072 (local address 127.0.97.60:37319)
0620 03:11:20.527571 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.527578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.527836 (+ 258us) server_negotiation.cc:407] Connection header received
0620 03:11:20.528029 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.528035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.528180 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.528303 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:20.528667 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.529340 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.530609 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.531002 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.531132 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:20.531654 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:20.532153 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:20.532313 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:20.532365 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 03:11:20.532439 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:11:20.545675 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:20.538177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.0.97.61:46697)
0620 03:11:20.538479 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.538488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.539097 (+ 609us) server_negotiation.cc:407] Connection header received
0620 03:11:20.539396 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.539402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.539477 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.539574 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:20.540152 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.541269 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.543492 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.544123 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.544242 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:20.544915 (+ 673us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:20.545041 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:20.545102 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:20.545315 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 03:11:20.545392 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260620 03:11:20.546016 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:20.539232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.0.97.60:37319)
0620 03:11:20.539432 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.539438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.540722 (+ 1284us) server_negotiation.cc:407] Connection header received
0620 03:11:20.540889 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.540898 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.540966 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.541056 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:20.541586 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.542323 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.544223 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.544726 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.544856 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:20.545234 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:20.545365 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:20.545446 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:20.545671 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 03:11:20.545747 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 03:11:20.546308 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:11:20.538172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39890 (local address 127.0.97.62:46521)
0620 03:11:20.538439 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 03:11:20.538448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:11:20.539196 (+ 748us) server_negotiation.cc:407] Connection header received
0620 03:11:20.539476 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:11:20.539484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:11:20.539740 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:11:20.539926 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:11:20.540642 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.541683 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.543770 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:11:20.544206 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:11:20.544334 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:11:20.544986 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:11:20.545432 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:11:20.545516 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 03:11:20.545926 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 03:11:20.546015 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 03:11:20.550715 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 23191
I20260620 03:11:20.551815 23253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:11:20.661164 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 23191
I20260620 03:11:20.682399 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 23265
I20260620 03:11:20.683517 23327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:11:20.789139 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 23265
I20260620 03:11:20.809295 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5VLrcV/build/asan/bin/kudu pid 23339
I20260620 03:11:20.810361 23401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:11:20.915323 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5VLrcV/build/asan/bin/kudu with pid 23339
2026-06-20T03:11:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4273 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8641 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (360510 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST